| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9 to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4 to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-2f453-f98d689d-r6rr4 to ip-10-0-130-199.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl to ip-10-0-143-198.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-68984b9498-l98dp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-e0026-646bcd588f-djfrt to ip-10-0-130-199.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx to ip-10-0-141-190.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-6946b55967 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-6946b55967 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-6946b55967-dt4pf | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-585bd9bcc8 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-585bd9bcc8-9d4gz | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-585bd9bcc8 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-585bd9bcc8-5mhzk | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-b7ffd-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
AddedInterface |
Add eth0 [10.134.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
FailedMount |
MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" : failed to sync configmap cache: timed out waiting for the condition | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
AddedInterface |
Add eth0 [10.134.0.17/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : failed to sync secret cache: timed out waiting for the condition | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" in 3.408s (3.408s including waiting). Image size: 306813343 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" in 2.931s (2.931s including waiting). Image size: 306813343 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 15.998s (15.998s including waiting). Image size: 1583067017 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 16.025s (16.025s including waiting). Image size: 1583067017 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.594s (2.594s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 16.205s (16.205s including waiting). Image size: 1583067017 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 16.211s (16.211s including waiting). Image size: 1583067017 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.646s (2.646s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.761s (2.761s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.924s (2.924s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" in 2.46s (2.46s including waiting). Image size: 243360834 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Created |
Created container: kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b7ffd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b7ffd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.18:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.17:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b7ffd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b7ffd |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-b7ffd] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-585bd9bcc8 |
SuccessfulDelete |
Deleted pod: isvc-canary-rollback-predictor-585bd9bcc8-5mhzk | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b7ffd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-rollback-predictor-585bd9bcc8 from 2 to 1 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b7ffd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-5mhzk |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.18:8643/healthz": dial tcp 10.134.0.18:8643: connect: connection refused | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 1 to 2 | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-585bd9bcc8 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-585bd9bcc8-9fltn | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
AddedInterface |
Add eth0 [10.134.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-6946b55967-dt4pf |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.17:8643/healthz": dial tcp 10.134.0.17:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-2f453 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-2f453": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-2f453": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-2f453-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-2f453-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
AddedInterface |
Add eth0 [10.134.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-2f453-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-2f453-predictor-cbd89768c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-2f453-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-2f453-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.928s (18.928s including waiting). Image size: 1436472579 bytes. | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-2f453-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-2f453-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-2f453 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-2f453] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-2f453 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9d4gz |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.19:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-585bd9bcc8-9fltn |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.19:8643/healthz": dial tcp 10.134.0.19:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-5f87c564d7 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-5f87c564d7-c88ht | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-5f87c564d7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5d5b899946 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-5d5b899946-b7bt5 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5d5b899946 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-5d5b899946-mhxmz | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-5d5b899946 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-2f453 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-2f453 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-2f453] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-2f453-f98d689d |
SuccessfulCreate |
Created pod: model-chainer-raw-2f453-f98d689d-r6rr4 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-2f453-serving-cert" not found | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-2f453 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-2f453] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-2f453 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-2f453-f98d689d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:341d7d999371c3925a2bf1dab18a51cbb6b284ad1c310340df58f19b3f00d99f" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-2f453-f98d689d-r6rr4 |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Created |
Created container: model-chainer-raw-2f453 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Started |
Started container model-chainer-raw-2f453 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:341d7d999371c3925a2bf1dab18a51cbb6b284ad1c310340df58f19b3f00d99f" in 1.893s (1.893s including waiting). Image size: 214866500 bytes. | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-2f453-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-2f453-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-2f453-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-2f453-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Killing |
Stopping container model-chainer-raw-2f453 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-e0026-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-e0026-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-e0026-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.20:8643/healthz": dial tcp 10.134.0.20:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.575s (19.575s including waiting). Image size: 1436472579 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-2f453-f98d689d-r6rr4 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-e0026-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-e0026-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-e0026-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-e0026-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-stop-predictor-5d5b899946 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5d5b899946 |
SuccessfulDelete |
Deleted pod: isvc-canary-stop-predictor-5d5b899946-b7bt5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-b7bt5 |
Killing |
Stopping container kube-rbac-proxy | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5d5b899946-mhxmz |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5f87c564d7-c88ht |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-81daee-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-81daee |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-81daee": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-81daee-predictor-865f44d78 |
SuccessfulCreate |
Created pod: isvc-primary-81daee-predictor-865f44d78-xlh89 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-81daee-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-81daee-predictor-865f44d78 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-81daee |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-81daee": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-81daee": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-e0026 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-e0026 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-e0026] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-e0026 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-e0026] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-e0026 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-e0026 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-e0026-646bcd588f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-e0026-646bcd588f |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-e0026-646bcd588f-djfrt | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-e0026 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-e0026] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Created |
Created container: model-chainer-raw-hpa-e0026 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Started |
Started container model-chainer-raw-hpa-e0026 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:341d7d999371c3925a2bf1dab18a51cbb6b284ad1c310340df58f19b3f00d99f" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-e0026-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-e0026-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-e0026-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-e0026-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-91b5b-predictor-598d8989f5 |
SuccessfulCreate |
Created pod: message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-91b5b-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-91b5b-predictor-598d8989f5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Killing |
Stopping container model-chainer-raw-hpa-e0026 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-91b5b-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.087s (1.087s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-91b5b-predictor-578588fc8f |
SuccessfulCreate |
Created pod: isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-91b5b |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-91b5b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-91b5b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-91b5b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-91b5b-predictor-578588fc8f from 0 to 1 | |
| (x6) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-91b5b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-91b5b |
InferenceServiceReady |
InferenceService [message-dumper-raw-91b5b] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-81daee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-81daee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Started |
Started container agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-e0026-646bcd588f-djfrt |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-91b5b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-91b5b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-91b5b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-91b5b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-81daee |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-81daee |
InferenceServiceReady |
InferenceService [isvc-primary-81daee] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-91b5b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-91b5b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-81daee-predictor-5b59c775bf |
SuccessfulCreate |
Created pod: isvc-secondary-81daee-predictor-5b59c775bf-rbtsw | |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-81daee-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-81daee-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-81daee-predictor-5b59c775bf from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-81daee |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-81daee": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-81daee": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-81daee |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-81daee": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-81daee-predictor-5b59c775bf-rbtsw |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_kserve-ci-e2e-test(84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd) | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-81daee |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-81daee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-81daee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-81daee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-81daee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-91b5b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-91b5b |
InferenceServiceReady |
InferenceService [isvc-logger-raw-91b5b] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-81daee-predictor-865f44d78-xlh89 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-c46f26-predictor-688947fcf7 |
SuccessfulCreate |
Created pod: isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-c46f26-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-c46f26-predictor-688947fcf7 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-c46f26-predictor-serving-cert" not found |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-c46f26 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-c46f26": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-c46f26": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-c46f26 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-c46f26": the object has been modified; please apply your changes to the latest version and try again | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-91b5b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-91b5b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-1efdd-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-91b5b-predictor-598d8989f5-8jv89 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-1efdd-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-c46f26 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-c46f26-predictor-688947fcf7-2qj57 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_kserve-ci-e2e-test(d9f0f324-d328-4c4b-9646-b3011589817c) | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Created |
Created container: kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcbfb |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-fcbfb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-fcbfb": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-fcbfb-predictor-58ccc8ff55 |
SuccessfulCreate |
Created pod: raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-fcbfb-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-fcbfb-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-fcbfb-predictor-58ccc8ff55 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcbfb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcbfb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1efdd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1efdd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcbfb |
InferenceServiceReady |
InferenceService [raw-sklearn-fcbfb] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcbfb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcbfb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcbfb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aa327 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-aa327": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-aa327": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-aa327-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aa327-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aa327-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1efdd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1efdd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-1efdd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-1efdd |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-1efdd] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-579784d7d7 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-579784d7d7-wfkvd | |
| (x19) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-68984b9498 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-68984b9498-rd2x9 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-68984b9498 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-68984b9498-l98dp | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-68984b9498 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-579784d7d7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-68984b9498-rd2x9 |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-68984b9498-l98dp |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aa327-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aa327-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aa327 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-aa327] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aa327 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-68984b9498 |
SuccessfulDelete |
Deleted pod: isvc-canary-create-predictor-68984b9498-rd2x9 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Killing |
Stopping container kube-rbac-proxy | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-create-predictor-68984b9498 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-rd2x9 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-68984b9498-l98dp |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-579784d7d7-wfkvd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-cb75746f8 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-cb75746f8-grg87 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-c6f7b798f |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-cb75746f8 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-cb75746f8-zgvp4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-cb75746f8 from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-cb75746f8-grg87 |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-cb75746f8 |
SuccessfulDelete |
Deleted pod: isvc-canary-promote-predictor-cb75746f8-grg87 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-promote-predictor-cb75746f8 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-grg87 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-c6f7b798f |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-dllbl | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-cb75746f8-zgvp4 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Started |
Started container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-dllbl |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Pulled |
Container image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-6b79574594 |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-6b79574594-9swqs | |
| (x6) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-6b79574594 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" in 2.443s (2.443s including waiting). Image size: 243360834 bytes. | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-rqnx4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-6b79574594-9swqs |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |