| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-57a06-58698db664-42hgl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-p9x7s to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5 to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5 to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cnzwk to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 to ip-10-0-134-49.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm to ip-10-0-135-179.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6b8c75c7b9 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 0 to 2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-d953e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6b8c75c7b9 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-d953e |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-d953e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-d953e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-d953e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-d953e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-7b8b964c7b |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-7b8b964c7b from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
AddedInterface |
Add eth0 [10.132.0.16/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" in 2.898s (2.898s including waiting). Image size: 308174258 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" in 3.388s (3.388s including waiting). Image size: 308174258 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 16.251s (16.251s including waiting). Image size: 1583203731 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 15.99s (15.99s including waiting). Image size: 1583203731 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 13.306s (13.306s including waiting). Image size: 1583203731 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 13.362s (13.362s including waiting). Image size: 1583203731 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.267s (2.267s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.282s (2.282s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:bc0968532a374fa6dea5e41a2aa72ef7623bc7b67c9c3e951482b3b25d6ace5c" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.855s (2.855s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.624s (2.624s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:bc0968532a374fa6dea5e41a2aa72ef7623bc7b67c9c3e951482b3b25d6ace5c" in 2.302s (2.302s including waiting). Image size: 240302165 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Started |
Started container agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-d953e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-d953e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-d953e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-d953e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.16:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-d953e |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-d953e] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-d953e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6b8c75c7b9 |
SuccessfulDelete |
Deleted pod: isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-57a06-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-57a06-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
AddedInterface |
Add eth0 [10.132.0.17/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.16:8643/healthz": dial tcp 10.132.0.16:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Started |
Started container kube-rbac-proxy | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6b8c75c7b9 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
AddedInterface |
Add eth0 [10.132.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-d953e-predictor-7584c9b5d8-fjsqd |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 20.399s (20.399s including waiting). Image size: 1435742979 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-57a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-57a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-57a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-57a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-57a06 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-57a06 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-57a06] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-57a06 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-57a06 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-57a06] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.18:8643/healthz": dial tcp 10.132.0.18:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-57a06 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-57a06-58698db664 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5cdcf6545 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-5cdcf6545-hppnb | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-57a06 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-57a06] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-57a06-58698db664 |
SuccessfulCreate |
Created pod: model-chainer-raw-57a06-58698db664-42hgl | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-596664c988 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-596664c988 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-596664c988-4sbx5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-5cdcf6545 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-57a06-58698db664-42hgl |
AddedInterface |
Add eth0 [10.132.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:f6d52988e64cd8a9a21f609c0c944dcdc6613df8ea74ac8dfd85c52003a9bc37" | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5cdcf6545 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-5cdcf6545-qsl6n | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
AddedInterface |
Add eth0 [10.132.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Created |
Created container: model-chainer-raw-57a06 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:f6d52988e64cd8a9a21f609c0c944dcdc6613df8ea74ac8dfd85c52003a9bc37" in 2.133s (2.133s including waiting). Image size: 211751512 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Started |
Started container model-chainer-raw-57a06 | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-57a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-57a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-57a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-57a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Killing |
Stopping container model-chainer-raw-57a06 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-68381-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-68381-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-68381-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.17:8643/healthz": dial tcp 10.132.0.17:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-57a06-predictor-84958bdbb4-dfbmz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.17:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.992s (18.992s including waiting). Image size: 1435742979 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-57a06-58698db664-42hgl |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-68381-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-68381-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-68381-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-68381-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.20:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-stop-predictor-5cdcf6545 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5cdcf6545 |
SuccessfulDelete |
Deleted pod: isvc-canary-stop-predictor-5cdcf6545-qsl6n | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Killing |
Stopping container kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-68381 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-68381 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-68381] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Killing |
Stopping container kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-qsl6n |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-596664c988-4sbx5 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5cdcf6545-hppnb |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.20:8643/healthz": dial tcp 10.132.0.20:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-68381 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-68381] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-68381 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
AddedInterface |
Add eth0 [10.132.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-a7cd73 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a7cd73": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-a7cd73 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-a7cd73": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a7cd73": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-a7cd73-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-a7cd73-predictor-6887dfdd9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-a7cd73-predictor-6887dfdd9 |
SuccessfulCreate |
Created pod: isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-68381-695f48ffc |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-68381-695f48ffc-xdcbm | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-68381 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-68381-695f48ffc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Started |
Started container model-chainer-raw-hpa-68381 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Created |
Created container: model-chainer-raw-hpa-68381 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:f6d52988e64cd8a9a21f609c0c944dcdc6613df8ea74ac8dfd85c52003a9bc37" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
AddedInterface |
Add eth0 [10.132.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-68381 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-68381] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-68381-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-68381-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-68381-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-68381-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-7b2ba-predictor-5dd4c89768 |
SuccessfulCreate |
Created pod: message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Killing |
Stopping container model-chainer-raw-hpa-68381 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-7b2ba-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-7b2ba-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-7b2ba-predictor-5dd4c89768 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
AddedInterface |
Add eth0 [10.134.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.032s (1.032s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-68381-predictor-66494c8d76-v7fw2 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-7b2ba |
InferenceServiceReady |
InferenceService [message-dumper-raw-7b2ba] is Ready | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-7b2ba |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-7b2ba-predictor-69456d97b5 |
SuccessfulCreate |
Created pod: isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-7b2ba-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-7b2ba-predictor-69456d97b5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7b2ba |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-7b2ba": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-7b2ba": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
AddedInterface |
Add eth0 [10.134.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:bc0968532a374fa6dea5e41a2aa72ef7623bc7b67c9c3e951482b3b25d6ace5c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Created |
Created container: agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-68381-695f48ffc-xdcbm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-a7cd73-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7b2ba-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7b2ba-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-a7cd73-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7b2ba-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7b2ba-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7b2ba-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7b2ba-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-a7cd73 |
InferenceServiceReady |
InferenceService [isvc-primary-a7cd73] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-a7cd73 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-a7cd73-predictor-676bb47b74 |
SuccessfulCreate |
Created pod: isvc-secondary-a7cd73-predictor-676bb47b74-db44n | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-a7cd73-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-a7cd73-predictor-676bb47b74 from 0 to 1 | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-a7cd73-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-a7cd73 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a7cd73": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-a7cd73 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-a7cd73": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a7cd73": the object has been modified; please apply your changes to the latest version and try again | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-a7cd73-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-a7cd73-predictor-676bb47b74-db44n |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-a7cd73-predictor-676bb47b74-db44n_kserve-ci-e2e-test(bcf18be6-1954-4a5e-8578-0a7551454bc5) | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-a7cd73 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7b2ba |
InferenceServiceReady |
InferenceService [isvc-logger-raw-7b2ba] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7b2ba |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-a7cd73-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-a7cd73-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7b2ba-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7b2ba-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-da0909 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-da0909": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-da0909 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-da0909": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-da0909": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-da0909-predictor-f56d96658 |
SuccessfulCreate |
Created pod: isvc-init-fail-da0909-predictor-f56d96658-7q88j | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-da0909-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-da0909-predictor-f56d96658 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-603e1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7b2ba-predictor-5dd4c89768-4vsqc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
Created |
Created container: storage-initializer |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-da0909 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-da0909-predictor-f56d96658-7q88j |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-103f5-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-103f5-predictor-6594b785b8 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-103f5 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-103f5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-103f5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-103f5-predictor-6594b785b8 |
SuccessfulCreate |
Created pod: raw-sklearn-103f5-predictor-6594b785b8-857v5 | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-103f5-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7b2ba-predictor-69456d97b5-skdpj |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-603e1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-603e1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-103f5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-103f5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-103f5 |
InferenceServiceReady |
InferenceService [raw-sklearn-103f5] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-103f5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-103f5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-103f5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-103f5-predictor-6594b785b8-857v5 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bfe4d |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-bfe4d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-bfe4d": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
AddedInterface |
Add eth0 [10.134.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-bfe4d-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-bfe4d-predictor-5db4699b8d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bfe4d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-bfe4d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bfe4d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bfe4d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-603e1 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-603e1] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-603e1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-603e1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-603e1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Killing |
Stopping container kserve-container | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-6474b875b9 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-6474b875b9-6qfbj | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-6474b875b9 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-6474b875b9-p9x7s | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-6474b875b9-p9x7s |
AddedInterface |
Add eth0 [10.134.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-6474b875b9 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-6474b875b9-6qfbj |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-96f77bdbd |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-96f77bdbd-h8tpl | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-96f77bdbd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bfe4d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bfe4d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bfe4d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bfe4d |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-bfe4d] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bfe4d-predictor-5db4699b8d-2qj6f |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-create-predictor-6474b875b9 from 2 to 1 | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-6474b875b9 |
SuccessfulDelete |
Deleted pod: isvc-canary-create-predictor-6474b875b9-p9x7s | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-p9x7s |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-6474b875b9-6qfbj |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-96f77bdbd-h8tpl |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-5b4b466b66 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-7cf78f884f from 0 to 2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-5b4b466b66 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-7cf78f884f |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-7cf78f884f-fxk4m | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-7cf78f884f |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-7cf78f884f-vbt2p | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
AddedInterface |
Add eth0 [10.134.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-7cf78f884f |
SuccessfulDelete |
Deleted pod: isvc-canary-promote-predictor-7cf78f884f-vbt2p | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-promote-predictor-7cf78f884f from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-vbt2p |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-5b4b466b66 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Killing |
Stopping container kube-rbac-proxy | |
| (x23) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-5b4b466b66 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-7cf78f884f-fxk4m |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5b4b466b66-pbxnn |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Pulled |
Container image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" already present on machine | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
AddedInterface |
Add eth0 [10.134.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-cnzwk | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
| (x6) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-778ff5f68f |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-778ff5f68f from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
AddedInterface |
Add eth0 [10.134.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:051af6a7c6b8b08a9503c479397f58819d99aa2694c94c5895c362bcc590737e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:bc0968532a374fa6dea5e41a2aa72ef7623bc7b67c9c3e951482b3b25d6ace5c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Created |
Created container: agent | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Killing |
Stopping container agent | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-cnzwk |
Killing |
Stopping container kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-778ff5f68f-45rhd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused |