| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-e3535-66db6d66bd-2f6xn to ip-10-0-133-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-wvx2q to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-p87sf to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-b3e077-predictor-556784fdbf-vrbrt to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-c9kz2 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx to ip-10-0-133-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85 to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-m9vwm to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-7s427 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-2wc45 to ip-10-0-131-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 to ip-10-0-133-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl to ip-10-0-138-91.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-76998db6fd-mcm7j to ip-10-0-133-250.ec2.internal | ||
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-0d68f |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-0d68f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-0d68f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-0d68f |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-0d68f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-0d68f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
AddedInterface |
Add eth0 [10.133.0.16/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-76998db6fd |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-76998db6fd-mcm7j | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
AddedInterface |
Add eth0 [10.133.0.17/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-548b9d8f79 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-76998db6fd from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-76998db6fd |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-76998db6fd-7s427 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-548b9d8f79 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" in 3.035s (3.035s including waiting). Image size: 308281769 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" in 3.319s (3.319s including waiting). Image size: 308281769 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" in 2.988s (2.988s including waiting). Image size: 308281769 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" in 13.193s (13.193s including waiting). Image size: 1580720527 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" in 13.262s (13.262s including waiting). Image size: 1580720527 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.223s (2.223s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:f4032031248ea217fb336a930cee0b43dfad133eb5dfde428dddf7e35cd320a4" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.2s (2.2s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" in 15.739s (15.739s including waiting). Image size: 1580720527 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" in 15.749s (15.749s including waiting). Image size: 1580720527 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:f4032031248ea217fb336a930cee0b43dfad133eb5dfde428dddf7e35cd320a4" in 2.351s (2.351s including waiting). Image size: 243416141 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.513s (2.513s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.483s (2.483s including waiting). Image size: 211946088 bytes. | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-0d68f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-0d68f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.17:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-0d68f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-0d68f |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-0d68f] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-76998db6fd |
SuccessfulDelete |
Deleted pod: isvc-canary-rollback-predictor-76998db6fd-7s427 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-rollback-predictor-76998db6fd from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Killing |
Stopping container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-0d68f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-0d68f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-7s427 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.17:8643/healthz": dial tcp 10.133.0.17:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-76998db6fd |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-76998db6fd-p87sf | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-76998db6fd from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
AddedInterface |
Add eth0 [10.133.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-e3535 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-e3535": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-e3535": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.16:8643/healthz": dial tcp 10.133.0.16:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-548b9d8f79-tqdzc |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.16:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-e3535-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-e3535-predictor-778c796d4f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-e3535-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-e3535-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
AddedInterface |
Add eth0 [10.133.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Started |
Started container kserve-container | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.004s (18.004s including waiting). Image size: 1332293378 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Created |
Created container: kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-e3535-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-e3535-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-e3535-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-e3535-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.19:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.18:8643/healthz": dial tcp 10.133.0.18:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-p87sf |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-76998db6fd-mcm7j |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-e3535 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-e3535 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-e3535] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-86f55c65d7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-79f67b4d78 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-79f67b4d78-v7v85 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
AddedInterface |
Add eth0 [10.133.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-86f55c65d7 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-79f67b4d78 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-79f67b4d78-2wc45 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-79f67b4d78 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
AddedInterface |
Add eth0 [10.132.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Started |
Started container storage-initializer | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-e3535 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-e3535 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-e3535] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-e3535-66db6d66bd |
SuccessfulCreate |
Created pod: model-chainer-raw-e3535-66db6d66bd-2f6xn | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-e3535 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-e3535-66db6d66bd from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-e3535 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-e3535] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-e3535-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:ca6150683b88d0bc774f8e42803fb5db7032d026ca2a7ea35068f270ce0317f1" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Started |
Started container model-chainer-raw-e3535 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Created |
Created container: model-chainer-raw-e3535 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:ca6150683b88d0bc774f8e42803fb5db7032d026ca2a7ea35068f270ce0317f1" in 1.64s (1.64s including waiting). Image size: 214921807 bytes. | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-e3535-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-e3535-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-e3535-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-e3535-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Killing |
Stopping container model-chainer-raw-e3535 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-bd0a0-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
AddedInterface |
Add eth0 [10.133.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-e3535-predictor-66c6766f6c-whwv8 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.19:8643/healthz": dial tcp 10.133.0.19:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.68s (18.68s including waiting). Image size: 1332293378 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-e3535-66db6d66bd-2f6xn |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.20:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-stop-predictor-79f67b4d78 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-79f67b4d78 |
SuccessfulDelete |
Deleted pod: isvc-canary-stop-predictor-79f67b4d78-2wc45 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Killing |
Stopping container kube-rbac-proxy | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-2wc45 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.20:8643/healthz": dial tcp 10.133.0.20:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-79f67b4d78-v7v85 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
multus |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
AddedInterface |
Add eth0 [10.132.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-b3e077-predictor-54d44d9966 |
SuccessfulCreate |
Created pod: isvc-primary-b3e077-predictor-54d44d9966-kjzpf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-b3e077-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-b3e077-predictor-54d44d9966 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-b3e077 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-b3e077": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-b3e077": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-b3e077 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-b3e077": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Started |
Started container storage-initializer | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-bd0a0 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-bd0a0 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-bd0a0] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Created |
Created container: kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-bd0a0 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-bd0a0 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-bd0a0] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-bd0a0-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-bd0a0 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-bd0a0-57bc6b4ffd from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-bd0a0 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-bd0a0] is Ready | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:ca6150683b88d0bc774f8e42803fb5db7032d026ca2a7ea35068f270ce0317f1" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Created |
Created container: model-chainer-raw-hpa-bd0a0 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Started |
Started container model-chainer-raw-hpa-bd0a0 | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-d98d3-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-d98d3-predictor-5887996fd4 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Killing |
Stopping container model-chainer-raw-hpa-bd0a0 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-b3e077-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-b3e077-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-d98d3-predictor-5887996fd4 |
SuccessfulCreate |
Created pod: message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d98d3 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-d98d3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-d98d3": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-d98d3-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-bd0a0-predictor-94467dbb5-ks9c6 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.156s (1.156s including waiting). Image size: 14813193 bytes. | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-bd0a0-predictor-55994c575-qzd7n |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.21:8643/healthz": dial tcp 10.133.0.21:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d98d3 |
InferenceServiceReady |
InferenceService [message-dumper-raw-d98d3] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d98d3 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d98d3-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d98d3-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-d98d3-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-d98d3-predictor-769b867bd6 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d98d3 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-d98d3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-d98d3": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-d98d3-predictor-769b867bd6 |
SuccessfulCreate |
Created pod: isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-d98d3-predictor-serving-cert" not found |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:f4032031248ea217fb336a930cee0b43dfad133eb5dfde428dddf7e35cd320a4" | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-bd0a0-57bc6b4ffd-sqwsx |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:f4032031248ea217fb336a930cee0b43dfad133eb5dfde428dddf7e35cd320a4" in 2.808s (2.808s including waiting). Image size: 243416141 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Started |
Started container agent | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d98d3-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d98d3-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-b3e077 |
InferenceServiceReady |
InferenceService [isvc-primary-b3e077] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-b3e077 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d98d3-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d98d3-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-b3e077-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-b3e077-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-b3e077-predictor-556784fdbf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-b3e077-predictor-serving-cert" not found | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-b3e077-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-b3e077 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-b3e077": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-b3e077": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-b3e077 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-b3e077": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-b3e077-predictor-556784fdbf |
SuccessfulCreate |
Created pod: isvc-secondary-b3e077-predictor-556784fdbf-vrbrt | |
kserve-ci-e2e-test |
multus |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-b3e077 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-b3e077-predictor-556784fdbf-vrbrt |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-b3e077-predictor-556784fdbf-vrbrt_kserve-ci-e2e-test(2b52d1d0-69cb-4f87-9d24-d68b8a90dcef) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-b3e077-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-b3e077-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-b3e077-predictor-54d44d9966-kjzpf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc |
SuccessfulCreate |
Created pod: isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-4bf2b4-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-4bf2b4 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-4bf2b4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-4bf2b4 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-4bf2b4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-4bf2b4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-4bf2b4-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
AddedInterface |
Add eth0 [10.132.0.41/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-4bf2b4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
Started |
Started container storage-initializer |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d98d3 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d98d3 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-d98d3] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74 |
Killing |
Stopping container storage-initializer | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d98d3-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d98d3-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-03136-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-03136-predictor-84fcd88955 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-03136-predictor-84fcd88955 |
SuccessfulCreate |
Created pod: raw-sklearn-03136-predictor-84fcd88955-hs42x | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-03136 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-03136": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-03136": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-a62a7-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Created |
Created container: storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d98d3-predictor-769b867bd6-tdf2w |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a62a7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a62a7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-03136-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-03136-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-03136 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-03136 |
InferenceServiceReady |
InferenceService [raw-sklearn-03136] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-03136-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-03136-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-03136-predictor-84fcd88955-hs42x |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17b5f |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-17b5f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-17b5f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-17b5f-predictor-7bb86784c |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
AddedInterface |
Add eth0 [10.132.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-17b5f-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-17b5f-predictor-7bb86784c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17b5f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17b5f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-a62a7 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-a62a7] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-a62a7 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a62a7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a62a7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Killing |
Stopping container kube-rbac-proxy | |
| (x19) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-54dc6cb966 from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-54dc6cb966 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-54dc6cb966-c9kz2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-54dc6cb966 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-54dc6cb966-dsmct | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-6f8d77bbc9 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-6f8d77bbc9 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-54dc6cb966-dsmct |
AddedInterface |
Add eth0 [10.132.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a62a7-predictor-5dbfb56d49-hxwq4 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17b5f |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-17b5f] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
AddedInterface |
Add eth0 [10.132.0.44/23] from ovn-kubernetes | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17b5f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Killing |
Stopping container kube-rbac-proxy | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17b5f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17b5f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Killing |
Stopping container kserve-container | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-create-predictor-54dc6cb966 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-54dc6cb966 |
SuccessfulDelete |
Deleted pod: isvc-canary-create-predictor-54dc6cb966-c9kz2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-c9kz2 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-54dc6cb966-dsmct |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-6cbdffdfdb |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-6cbdffdfdb-zbk5x | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-6cbdffdfdb from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
AddedInterface |
Add eth0 [10.132.0.46/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-698cc558fc |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-698cc558fc-r889r | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-698cc558fc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-6cbdffdfdb |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-6cbdffdfdb-wvx2q | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
AddedInterface |
Add eth0 [10.132.0.47/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-promote-predictor-6cbdffdfdb from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-6cbdffdfdb |
SuccessfulDelete |
Deleted pod: isvc-canary-promote-predictor-6cbdffdfdb-wvx2q | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-wvx2q |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-698cc558fc from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-698cc558fc |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-698cc558fc-m9vwm | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-6cbdffdfdb-zbk5x |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-r889r |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-698cc558fc-m9vwm |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-hhkjh | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
AddedInterface |
Add eth0 [10.132.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Pulled |
Container image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Created |
Created container: kserve-container | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-84b648dc8b from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-84b648dc8b |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:10dce2bd6581ed3398139cc0e491de5addc169f1b70e0e912dc9be169495c7fb" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1947" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:f4032031248ea217fb336a930cee0b43dfad133eb5dfde428dddf7e35cd320a4" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-hhkjh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-84b648dc8b-hj622 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |