| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-596476db66-z2fhv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44 to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-596476db66-ckjrl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-ckjrl to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6 to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5 to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-cx4bd to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-sxxms to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-dmgqr to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-6d2jt to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-7b7df-5957dcc4dd-rlrd4 to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-49w9v to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h to ip-10-0-142-4.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r to ip-10-0-140-43.ec2.internal | ||
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-a264e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-a264e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-7df6589c46 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-7df6589c46-cx4bd | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-b9d76bcd7 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-b9d76bcd7 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-7df6589c46 from 0 to 2 | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
AddedInterface |
Add eth0 [10.132.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-a264e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-7df6589c46 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-7df6589c46-t9fj5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-a264e |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-a264e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-a264e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
AddedInterface |
Add eth0 [10.132.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" in 3.329s (3.329s including waiting). Image size: 306739683 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" in 3.831s (3.831s including waiting). Image size: 306739683 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" in 16.089s (16.089s including waiting). Image size: 1580775854 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" in 16.058s (16.058s including waiting). Image size: 1580775854 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:0747765e9bfc84706177c8d04eece41800cf0a209415692e8ad8ead1a0eafce3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.148s (3.148s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.096s (3.096s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" in 16.138s (16.138s including waiting). Image size: 1580775854 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" in 16.144s (16.144s including waiting). Image size: 1580775854 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:0747765e9bfc84706177c8d04eece41800cf0a209415692e8ad8ead1a0eafce3" in 2.404s (2.404s including waiting). Image size: 240302214 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.446s (2.446s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.437s (2.437s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Started |
Started container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-a264e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-a264e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-a264e |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-a264e] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-a264e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-rollback-predictor-7df6589c46 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-t9fj5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-7df6589c46 |
SuccessfulDelete |
Deleted pod: isvc-canary-rollback-predictor-7df6589c46-t9fj5 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-a264e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-a264e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-7df6589c46 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8 |
Killing |
Stopping container kube-rbac-proxy | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-7df6589c46 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-7df6589c46-zw6z6 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-7b7df-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-7b7df-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-7b7df-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.20:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-a264e-predictor-7bc5f796dc-gxcgj |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.20:8643/healthz": dial tcp 10.132.0.20:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.517s (18.517s including waiting). Image size: 1435742980 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-7b7df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-7b7df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-7b7df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-7b7df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-7b7df |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-7b7df |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-7b7df] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-cx4bd |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-7df6589c46-zw6z6 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-596476db66 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-596476db66-z2fhv | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-75db96cc6c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-75db96cc6c |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-596476db66 from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-596476db66 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-596476db66-ckjrl | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-7b7df |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-596476db66-ckjrl |
AddedInterface |
Add eth0 [10.132.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-7b7df |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-7b7df] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-596476db66-z2fhv |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:72663b376a728834b8b7a0627c1f77261896d78389c8bb03de412e525cf47359" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-7b7df |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-7b7df] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-7b7df |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-7b7df-5957dcc4dd from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-7b7df-5957dcc4dd |
SuccessfulCreate |
Created pod: model-chainer-raw-7b7df-5957dcc4dd-rlrd4 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Started |
Started container model-chainer-raw-7b7df | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Created |
Created container: model-chainer-raw-7b7df | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:72663b376a728834b8b7a0627c1f77261896d78389c8bb03de412e525cf47359" in 2.007s (2.007s including waiting). Image size: 211751048 bytes. | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-7b7df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-7b7df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-7b7df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-7b7df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Killing |
Stopping container model-chainer-raw-7b7df | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-d5772-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-d5772-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-d5772-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.883s (19.883s including waiting). Image size: 1435742980 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-7b7df-5957dcc4dd-rlrd4 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-d5772-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-d5772-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-d5772-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-d5772-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-stop-predictor-596476db66 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-596476db66 |
SuccessfulDelete |
Deleted pod: isvc-canary-stop-predictor-596476db66-z2fhv | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-z2fhv |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-596476db66-ckjrl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-67d4dc |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-67d4dc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-67d4dc-predictor-774f887c4b |
SuccessfulCreate |
Created pod: isvc-primary-67d4dc-predictor-774f887c4b-h476t | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-67d4dc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-67d4dc-predictor-774f887c4b from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-67d4dc |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-67d4dc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-67d4dc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-d5772 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-d5772] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-d5772 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Created |
Created container: kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-d5772 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-d5772 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-d5772] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-d5772-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-d5772-75ffc6d89c |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-d5772 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-d5772-75ffc6d89c from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-d5772 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-d5772] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:72663b376a728834b8b7a0627c1f77261896d78389c8bb03de412e525cf47359" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Created |
Created container: model-chainer-raw-hpa-d5772 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Started |
Started container model-chainer-raw-hpa-d5772 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-d5772-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-d5772-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-d5772-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-d5772-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-0a0da |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-0a0da": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-0a0da": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-d5772-predictor-54794bf88c-ptxqp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-0a0da-predictor-c8fbfbc9d |
SuccessfulCreate |
Created pod: message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Killing |
Stopping container model-chainer-raw-hpa-d5772 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-0a0da-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-0a0da-predictor-c8fbfbc9d from 0 to 1 | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.247s (1.247s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Created |
Created container: kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-d5772-predictor-56c57b5777-jrznx |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-0a0da |
InferenceServiceReady |
InferenceService [message-dumper-raw-0a0da] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-0a0da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-67d4dc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-67d4dc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-0a0da |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-0a0da": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-0a0da": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-0a0da-predictor-64d6ff776d |
SuccessfulCreate |
Created pod: isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-0a0da-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-0a0da-predictor-64d6ff776d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:0747765e9bfc84706177c8d04eece41800cf0a209415692e8ad8ead1a0eafce3" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-d5772-75ffc6d89c-k2m2h |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-0a0da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-0a0da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-0a0da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-0a0da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-67d4dc |
InferenceServiceReady |
InferenceService [isvc-primary-67d4dc] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-67d4dc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-0a0da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-0a0da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-67d4dc |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-67d4dc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-67d4dc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-67d4dc-predictor-8548b55dd5 |
SuccessfulCreate |
Created pod: isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-67d4dc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-67d4dc-predictor-8548b55dd5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-67d4dc |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-67d4dc": the object has been modified; please apply your changes to the latest version and try again | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-67d4dc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-67d4dc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-67d4dc-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_kserve-ci-e2e-test(59a62353-6bd2-4021-b180-d3f3b3d03c25) | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-67d4dc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-67d4dc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-67d4dc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-67d4dc-predictor-774f887c4b-h476t |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-edaf65-predictor-6d87d9675c |
SuccessfulCreate |
Created pod: isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-edaf65-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-edaf65-predictor-6d87d9675c from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-edaf65 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-edaf65": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-edaf65": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-edaf65 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-edaf65": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
Created |
Created container: storage-initializer |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-edaf65 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-0a0da |
InferenceServiceReady |
InferenceService [isvc-logger-raw-0a0da] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-0a0da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-35411-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-35411-predictor-65bc89658c from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-35411 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-35411": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-35411": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-35411-predictor-65bc89658c |
SuccessfulCreate |
Created pod: raw-sklearn-35411-predictor-65bc89658c-dth4r | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Created |
Created container: storage-initializer | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-0a0da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-0a0da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-a284b-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-a284b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-a284b-predictor-684c656bd9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a284b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a284b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-0a0da-predictor-64d6ff776d-ccvv9 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-35411-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-35411-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-35411 |
InferenceServiceReady |
InferenceService [raw-sklearn-35411] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-35411 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-35411-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-35411-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-35411-predictor-65bc89658c-dth4r |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60b0d |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-60b0d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-60b0d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-60b0d-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-60b0d-predictor-55c6df5959 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-60b0d-predictor-55c6df5959 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Created |
Created container: kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60b0d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60b0d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-a284b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-a284b |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-a284b] is Ready | |
| (x10) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a284b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-a284b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60b0d |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-60b0d] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Killing |
Stopping container kserve-container | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60b0d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-7cfc7d4c75 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-65f7bb5954 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-65f7bb5954-5hl44 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-65f7bb5954 from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-65f7bb5954 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-65f7bb5954-dmgqr | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-7cfc7d4c75 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60b0d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Killing |
Stopping container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60b0d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Created |
Created container: kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-create-predictor-65f7bb5954 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-65f7bb5954 |
SuccessfulDelete |
Deleted pod: isvc-canary-create-predictor-65f7bb5954-5hl44 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-5hl44 |
Killing |
Stopping container kube-rbac-proxy | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Killing |
Stopping container kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-65f7bb5954-dmgqr |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.28:8643/healthz": dial tcp 10.132.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-8555cb9565 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-8555cb9565-975sd | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-fc78f5cb6 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-fc78f5cb6-65h2m | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-fc78f5cb6 from 0 to 2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-8555cb9565 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-fc78f5cb6 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-fc78f5cb6-49w9v | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Created |
Created container: kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-promote-predictor-fc78f5cb6 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-fc78f5cb6 |
SuccessfulDelete |
Deleted pod: isvc-canary-promote-predictor-fc78f5cb6-65h2m | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-65h2m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-8555cb9565 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-8555cb9565-sxxms | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Killing |
Stopping container kserve-container | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-8555cb9565 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-fc78f5cb6-49w9v |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-975sd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-8555cb9565-sxxms |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-cipn-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-6d2jt | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.257s (1.257s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-856f4c4b54 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
AddedInterface |
Add eth0 [10.132.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-856f4c4b54 |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:fda42133f9b1cd73575073071ee5583bebe6a773707fa25ca330879e0107d1d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1976" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:0747765e9bfc84706177c8d04eece41800cf0a209415692e8ad8ead1a0eafce3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-6d2jt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-856f4c4b54-grmbw |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused |