Time Namespace Component RelatedObject Reason Message

openshift-ingress

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c to ip-10-0-128-50.us-west-2.compute.internal

ai-tenants

e2e-extmod-a-8ecbc5

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-a-8ecbc5: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-extmod-b-1bf4bf

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-b-1bf4bf: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-extmod-b-59c311

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-b-59c311: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-extmod-b-951893

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-b-951893: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-worker-w0-092f8c

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-worker-w0-092f8c: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-worker-w1-ee1947

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-worker-w1-ee1947: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-worker-w2-fba151

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-worker-w2-fba151: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-worker-w3-82c724

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-worker-w3-82c724: MaasTenantConfig is not available to restore revocation operands

ai-tenants

e2e-worker-w4-50429e

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-worker-w4-50429e: MaasTenantConfig is not available to restore revocation operands

cert-manager

cert-manager-597b95cb84-d2kl5

Scheduled

Successfully assigned cert-manager/cert-manager-597b95cb84-d2kl5 to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5 to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv to ip-10-0-133-201.us-west-2.compute.internal

cert-manager

cert-manager-cainjector-5c57bbd4fc-x67vn

Scheduled

Successfully assigned cert-manager/cert-manager-cainjector-5c57bbd4fc-x67vn to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-processing-7bb9489676-khhp6

Scheduled

Successfully assigned openshift-ingress/payload-processing-7bb9489676-khhp6 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47 to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv to ip-10-0-128-50.us-west-2.compute.internal

cert-manager

cert-manager-webhook-5dbbbf75cc-zsl4r

Scheduled

Successfully assigned cert-manager/cert-manager-webhook-5dbbbf75cc-zsl4r to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

authorino-56c6c9ddd5-2f296

Scheduled

Successfully assigned kuadrant-system/authorino-56c6c9ddd5-2f296 to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54 to ip-10-0-128-50.us-west-2.compute.internal

llm

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

Scheduled

Successfully assigned llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dr574 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

payload-pre-processing-7b98bb678c-chld2

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-chld2 to ip-10-0-135-242.us-west-2.compute.internal

llm

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

Scheduled

Successfully assigned llm/e2e-distinct-simulated-kserve-7b998b79d9-mgqz5 to ip-10-0-135-242.us-west-2.compute.internal

kuadrant-system

authorino-57f7f8659f-mqhqz

Scheduled

Successfully assigned kuadrant-system/authorino-57f7f8659f-mqhqz to ip-10-0-128-50.us-west-2.compute.internal

llm

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

Scheduled

Successfully assigned llm/e2e-embedding-simulated-kserve-67bf594cd6-r4s25 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

router-default-7cfb798487-kbfzk

FailedScheduling

no nodes available to schedule pods

openshift-ingress

router-default-7cfb798487-kbfzk

FailedScheduling

no nodes available to schedule pods

openshift-ingress

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

router-default-7cfb798487-kbfzk

Scheduled

Successfully assigned openshift-ingress/router-default-7cfb798487-kbfzk to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4 to ip-10-0-135-242.us-west-2.compute.internal

llm

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

Scheduled

Successfully assigned llm/e2e-trlp-test-simulated-kserve-69b588897d-55lw7 to ip-10-0-135-242.us-west-2.compute.internal

kuadrant-system

authorino-5cb7d797b8-rjd4x

Scheduled

Successfully assigned kuadrant-system/authorino-5cb7d797b8-rjd4x to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

istiod-openshift-gateway-7876bfcb74-mzwml

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-mzwml to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pmscg to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

kube-auth-proxy-7659c7d778-65h9l

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-7659c7d778-65h9l to ip-10-0-135-242.us-west-2.compute.internal

llm

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

Scheduled

Successfully assigned llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

kube-auth-proxy-7659c7d778-lxgpd

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-7659c7d778-lxgpd to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-26xvk to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-2tbrd to ip-10-0-128-50.us-west-2.compute.internal

llm

echo-simulator-kserve-64b6f8f96c-tbfbx

Scheduled

Successfully assigned llm/echo-simulator-kserve-64b6f8f96c-tbfbx to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

limitador-operator-controller-manager-5778bb4449-jtvzt

Scheduled

Successfully assigned kuadrant-system/limitador-operator-controller-manager-5778bb4449-jtvzt to ip-10-0-133-201.us-west-2.compute.internal

kuadrant-system

authorino-6f7c4f5d87-28j47

Scheduled

Successfully assigned kuadrant-system/authorino-6f7c4f5d87-28j47 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9 to ip-10-0-128-50.us-west-2.compute.internal

openshift-jobset-operator

jobset-controller-manager-696bfff7cc-5rxhf

Scheduled

Successfully assigned openshift-jobset-operator/jobset-controller-manager-696bfff7cc-5rxhf to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Scheduled

Successfully assigned openshift-ingress/e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

Scheduled

Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-bvxs5 to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8 to ip-10-0-133-201.us-west-2.compute.internal

kuadrant-system

authorino-76f555cd65-p5s8m

Scheduled

Successfully assigned kuadrant-system/authorino-76f555cd65-p5s8m to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-dh692

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-dh692 to ip-10-0-135-242.us-west-2.compute.internal

opendatahub-operator-system

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Scheduled

Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-6ffdd4467b-f8hl6 to ip-10-0-133-201.us-west-2.compute.internal

opendatahub-operator-system

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Scheduled

Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-6ffdd4467b-cj98l to ip-10-0-128-50.us-west-2.compute.internal

opendatahub-operator-system

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Scheduled

Successfully assigned opendatahub-operator-system/opendatahub-operator-controller-manager-6ffdd4467b-5cfww to ip-10-0-135-242.us-west-2.compute.internal

opendatahub

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

Scheduled

Successfully assigned opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-98hr6 to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

test-curl-7370-aff15b

Scheduled

Successfully assigned opendatahub/test-curl-7370-aff15b to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

test-curl-7370-7bea52

Scheduled

Successfully assigned opendatahub/test-curl-7370-7bea52 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fd2nl to ip-10-0-133-201.us-west-2.compute.internal

kuadrant-system

authorino-operator-7b7bb4bc77-hshfp

Scheduled

Successfully assigned kuadrant-system/authorino-operator-7b7bb4bc77-hshfp to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

authorino-6f6944bcdd-9dzz8

Scheduled

Successfully assigned kuadrant-system/authorino-6f6944bcdd-9dzz8 to ip-10-0-133-201.us-west-2.compute.internal

ai-tenants

e2e-extmod-a-6d170d

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-a-6d170d: MaasTenantConfig is not available to restore revocation operands

openshift-jobset-operator

jobset-operator-674cd59dc5-mtjbm

Scheduled

Successfully assigned openshift-jobset-operator/jobset-operator-674cd59dc5-mtjbm to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

odh-model-controller-769875db56-n2wm8

Scheduled

Successfully assigned opendatahub/odh-model-controller-769875db56-n2wm8 to ip-10-0-135-242.us-west-2.compute.internal

opendatahub

model-serving-api-75bd89fdfc-tkm8m

Scheduled

Successfully assigned opendatahub/model-serving-api-75bd89fdfc-tkm8m to ip-10-0-133-201.us-west-2.compute.internal

opendatahub

maas-controller-5fb896bc89-fkk4g

Scheduled

Successfully assigned opendatahub/maas-controller-5fb896bc89-fkk4g to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

maas-controller-589f8697fb-5r6hk

Scheduled

Successfully assigned opendatahub/maas-controller-589f8697fb-5r6hk to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

llmisvc-controller-manager-869d5dcc5d-w452v

Scheduled

Successfully assigned opendatahub/llmisvc-controller-manager-869d5dcc5d-w452v to ip-10-0-133-201.us-west-2.compute.internal

opendatahub

llmisvc-controller-manager-6466c554d8-fbbwk

Scheduled

Successfully assigned opendatahub/llmisvc-controller-manager-6466c554d8-fbbwk to ip-10-0-133-201.us-west-2.compute.internal

opendatahub

kserve-module-controller-manager-7566cf5fc8-qhqnk

Scheduled

Successfully assigned opendatahub/kserve-module-controller-manager-7566cf5fc8-qhqnk to ip-10-0-128-50.us-west-2.compute.internal

opendatahub

kserve-controller-manager-776cc66d47-rcb44

Scheduled

Successfully assigned opendatahub/kserve-controller-manager-776cc66d47-rcb44 to ip-10-0-133-201.us-west-2.compute.internal

opendatahub

ai-gateway-controller-7cd9b8cccb-7bml5

Scheduled

Successfully assigned opendatahub/ai-gateway-controller-7cd9b8cccb-7bml5 to ip-10-0-135-242.us-west-2.compute.internal

opendatahub

ai-gateway-controller-5d8dddb676-jcvfc

Scheduled

Successfully assigned opendatahub/ai-gateway-controller-5d8dddb676-jcvfc to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-fms7t to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-hgwbk to ip-10-0-128-50.us-west-2.compute.internal

models-as-a-service

payload-processing-external-model-df4cf9975-hk75w

Scheduled

Successfully assigned models-as-a-service/payload-processing-external-model-df4cf9975-hk75w to ip-10-0-128-50.us-west-2.compute.internal

models-as-a-service

payload-processing-external-model-8b89f9dc6-zbkqz

Scheduled

Successfully assigned models-as-a-service/payload-processing-external-model-8b89f9dc6-zbkqz to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-j6k8d to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-jx8n4 to ip-10-0-133-201.us-west-2.compute.internal

kuadrant-system

limitador-limitador-7c9b87d574-rqz2j

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-7c9b87d574-rqz2j to ip-10-0-133-201.us-west-2.compute.internal

models-as-a-service

payload-processing-external-model-7f5c68695b-7z22d

Scheduled

Successfully assigned models-as-a-service/payload-processing-external-model-7f5c68695b-7z22d to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-rjj92 to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

dns-operator-controller-manager-8557cbdb84-zttlw

Scheduled

Successfully assigned kuadrant-system/dns-operator-controller-manager-8557cbdb84-zttlw to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-smv2q to ip-10-0-133-201.us-west-2.compute.internal

models-as-a-service

payload-processing-external-model-7489856dc7-h45cw

Scheduled

Successfully assigned models-as-a-service/payload-processing-external-model-7489856dc7-h45cw to ip-10-0-128-50.us-west-2.compute.internal

models-as-a-service

payload-processing-external-model-5bcd5df458-7z9tf

Scheduled

Successfully assigned models-as-a-service/payload-processing-external-model-5bcd5df458-7z9tf to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-sr9mv to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-st9jq to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh to ip-10-0-133-201.us-west-2.compute.internal

openshift-ingress

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-v5h77 to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

limitador-limitador-74dfcfb894-hvj2g

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-74dfcfb894-hvj2g to ip-10-0-128-50.us-west-2.compute.internal

llm

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

Scheduled

Successfully assigned llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd to ip-10-0-128-50.us-west-2.compute.internal

kuadrant-system

kuadrant-operator-controller-manager-678d87766f-4hmfk

Scheduled

Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hmfk to ip-10-0-128-50.us-west-2.compute.internal

ai-tenants

e2e-extmod-a-5d4032

APIKeyCleanupSkipped

Skipping API-key cleanup for AITenant ai-tenants/e2e-extmod-a-5d4032: MaasTenantConfig is not available to restore revocation operands

kuadrant-system

kuadrant-console-plugin-66c4fb575d-hb9vr

Scheduled

Successfully assigned kuadrant-system/kuadrant-console-plugin-66c4fb575d-hb9vr to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

Scheduled

Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8 to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc to ip-10-0-135-242.us-west-2.compute.internal

llm

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

Scheduled

Successfully assigned llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv to ip-10-0-135-242.us-west-2.compute.internal

openshift-ingress

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Scheduled

Successfully assigned openshift-ingress/e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb to ip-10-0-128-50.us-west-2.compute.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-7cfb798487 from 0 to 1

openshift-ingress

replicaset-controller

router-default-7cfb798487

SuccessfulCreate

Created pod: router-default-7cfb798487-kbfzk

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts
(x6)

openshift-ingress

kubelet

router-default-7cfb798487-kbfzk

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
(x6)

openshift-ingress

kubelet

router-default-7cfb798487-kbfzk

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found

openshift-ingress

multus

router-default-7cfb798487-kbfzk

AddedInterface

Add eth0 [10.132.0.14/23] from ovn-kubernetes

openshift-ingress

kubelet

router-default-7cfb798487-kbfzk

Started

Container started

openshift-ingress

kubelet

router-default-7cfb798487-kbfzk

Created

Container created

openshift-ingress

kubelet

router-default-7cfb798487-kbfzk

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod

openshift-ingress

gateway_labeler_controller

maas-default-gateway

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7876bfcb74

SuccessfulCreate

Created pod: istiod-openshift-gateway-7876bfcb74-mzwml

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1

openshift-ingress

multus

istiod-openshift-gateway-7876bfcb74-mzwml

AddedInterface

Add eth0 [10.134.0.23/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.329s (4.329s including waiting). Image size: 185476477 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Created

Container created

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Started

Container started
(x2)

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:
(x2)

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 0 to 1

openshift-ingress

replicaset-controller

openshift-ai-inference-openshift-ai-inference-77b95bc695

SuccessfulCreate

Created pod: openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

openshift-ingress

service-controller

openshift-ai-inference-openshift-ai-inference

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

deployment-controller

openshift-ai-inference-openshift-ai-inference

ScalingReplicaSet

Scaled up replica set openshift-ai-inference-openshift-ai-inference-77b95bc695 from 0 to 1

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-v5h77

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

AddedInterface

Add eth0 [10.132.0.31/23] from ovn-kubernetes

openshift-ingress

multus

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

AddedInterface

Add eth0 [10.132.0.32/23] from ovn-kubernetes

openshift-ingress

kubelet

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"

openshift-ingress

service-controller

openshift-ai-inference-openshift-ai-inference

EnsuredLoadBalancer

Ensured load balancer

openshift-ingress

kubelet

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.711s (8.711s including waiting). Image size: 183555333 bytes.

openshift-ingress

kubelet

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.053s (9.053s including waiting). Image size: 183555333 bytes.

openshift-ingress

kubelet

openshift-ai-inference-openshift-ai-inference-77b95bc695-z75k8

Created

Container created

openshift-ingress

horizontal-pod-autoscaler

openshift-ai-inference-openshift-ai-inference

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

openshift-ai-inference-openshift-ai-inference

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

openshift-ai-inference-openshift-ai-inference

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

openshift-ai-inference-openshift-ai-inference

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

RequirementsNotMet

one or more requirements couldn't be found

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

RequirementsUnknown

requirements not yet checked

openshift-jobset-operator

replicaset-controller

jobset-operator-674cd59dc5

SuccessfulCreate

Created pod: jobset-operator-674cd59dc5-mtjbm

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

AllRequirementsMet

all requirements found, attempting install

openshift-jobset-operator

deployment-controller

jobset-operator

ScalingReplicaSet

Scaled up replica set jobset-operator-674cd59dc5 from 0 to 1
(x2)

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

InstallSucceeded

waiting for install components to report healthy
(x2)

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

InstallWaiting

installing: waiting for deployment jobset-operator to become ready: deployment "jobset-operator" not available: Deployment does not have minimum availability.

openshift-jobset-operator

multus

jobset-operator-674cd59dc5-mtjbm

AddedInterface

Add eth0 [10.132.0.40/23] from ovn-kubernetes

openshift-jobset-operator

kubelet

jobset-operator-674cd59dc5-mtjbm

Pulling

Pulling image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e"
(x2)

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

RequirementsNotMet

one or more requirements couldn't be found
(x2)

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

RequirementsUnknown

requirements not yet checked

openshift-jobset-operator

kubelet

jobset-operator-674cd59dc5-mtjbm

Started

Container started

openshift-jobset-operator

kubelet

jobset-operator-674cd59dc5-mtjbm

Created

Container created

openshift-jobset-operator

kubelet

jobset-operator-674cd59dc5-mtjbm

Pulled

Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9-operator@sha256:a258511e57b511c5d2a54fae2b8165669d677fb3a69796aa1c78d224a499033e" in 4.299s (4.299s including waiting). Image size: 224874686 bytes.

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

AllRequirementsMet

all requirements found, attempting install

opendatahub-operator-system

replicaset-controller

opendatahub-operator-controller-manager-6ffdd4467b

SuccessfulCreate

Created pod: opendatahub-operator-controller-manager-6ffdd4467b-cj98l

opendatahub-operator-system

replicaset-controller

opendatahub-operator-controller-manager-6ffdd4467b

SuccessfulCreate

Created pod: opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

opendatahub-operator-system

replicaset-controller

opendatahub-operator-controller-manager-6ffdd4467b

SuccessfulCreate

Created pod: opendatahub-operator-controller-manager-6ffdd4467b-5cfww

openshift-jobset-operator

operator-lifecycle-manager

jobset-operator.v1.0.1

InstallSucceeded

install strategy completed with no errors

opendatahub-operator-system

deployment-controller

opendatahub-operator-controller-manager

ScalingReplicaSet

Scaled up replica set opendatahub-operator-controller-manager-6ffdd4467b from 0 to 3

openshift-jobset-operator

openshift-jobset-operator

openshift-jobset-operator-lock

LeaderElection

jobset-operator-674cd59dc5-mtjbm_372104b0-6b08-4f64-b308-ec86d3702376 became leader

opendatahub-operator-system

multus

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

AddedInterface

Add eth0 [10.134.0.24/23] from ovn-kubernetes

opendatahub-operator-system

multus

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

AddedInterface

Add eth0 [10.133.0.17/23] from ovn-kubernetes

opendatahub-operator-system

multus

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

AddedInterface

Add eth0 [10.132.0.41/23] from ovn-kubernetes

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

InstallWaiting

Webhook install failed: conversionWebhook not ready

opendatahub-operator-system

operator-lifecycle-manager

install-v8zsv

AppliedWithWarnings

1 warning(s) generated during installation of operator "opendatahub-operator.v3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 4.468s (4.468s including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 4.305s (4.305s including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 4.242s (4.242s including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Created

Container created

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Created

Container created

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Created

Container created
(x3)

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

InstallSucceeded

waiting for install components to report healthy

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"
(x2)

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

InstallWaiting

installing: waiting for deployment opendatahub-operator-controller-manager to become ready: deployment "opendatahub-operator-controller-manager" not available: Deployment does not have minimum availability.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Pulling

Pulling image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2"

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Created

Container created

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 827ms (827ms including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Created

Container created

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-f8hl6

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 907ms (907ms including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Started

Container started

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-5cfww

Pulled

Successfully pulled image "quay.io/opendatahub/opendatahub-operator:v3.6.0-ea.2" in 882ms (882ms including waiting). Image size: 273709574 bytes.

opendatahub-operator-system

kubelet

opendatahub-operator-controller-manager-6ffdd4467b-cj98l

Created

Container created

opendatahub-operator-system

opendatahub-operator-controller-manager-6ffdd4467b-5cfww_4b8f1571-ae0e-4d74-99a0-9dc563d30c9e

07ed84f7.opendatahub.io

LeaderElection

opendatahub-operator-controller-manager-6ffdd4467b-5cfww_4b8f1571-ae0e-4d74-99a0-9dc563d30c9e became leader

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

RequirementsUnknown

requirements not yet checked

cert-manager

deployment-controller

cert-manager-cainjector

ScalingReplicaSet

Scaled up replica set cert-manager-cainjector-5c57bbd4fc from 0 to 1

cert-manager

deployment-controller

cert-manager

ScalingReplicaSet

Scaled up replica set cert-manager-597b95cb84 from 0 to 1
(x9)

cert-manager

replicaset-controller

cert-manager-cainjector-5c57bbd4fc

FailedCreate

Error creating: pods "cert-manager-cainjector-5c57bbd4fc-" is forbidden: error looking up service account cert-manager/cert-manager-cainjector: serviceaccount "cert-manager-cainjector" not found

kuadrant-system

replicaset-controller

dns-operator-controller-manager-8557cbdb84

SuccessfulCreate

Created pod: dns-operator-controller-manager-8557cbdb84-zttlw

kuadrant-system

deployment-controller

dns-operator-controller-manager

ScalingReplicaSet

Scaled up replica set dns-operator-controller-manager-8557cbdb84 from 0 to 1

cert-manager

multus

cert-manager-cainjector-5c57bbd4fc-x67vn

AddedInterface

Add eth0 [10.133.0.18/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

InstallWaiting

installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.

cert-manager

replicaset-controller

cert-manager-cainjector-5c57bbd4fc

SuccessfulCreate

Created pod: cert-manager-cainjector-5c57bbd4fc-x67vn

kuadrant-system

kubelet

dns-operator-controller-manager-8557cbdb84-zttlw

Pulling

Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a"

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

RequirementsNotMet

one or more requirements couldn't be found

cert-manager

kubelet

cert-manager-cainjector-5c57bbd4fc-x67vn

Pulling

Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"

kuadrant-system

multus

dns-operator-controller-manager-8557cbdb84-zttlw

AddedInterface

Add eth0 [10.132.0.43/23] from ovn-kubernetes

cert-manager

replicaset-controller

cert-manager-webhook-5dbbbf75cc

SuccessfulCreate

Created pod: cert-manager-webhook-5dbbbf75cc-zsl4r

cert-manager

deployment-controller

cert-manager-webhook

ScalingReplicaSet

Scaled up replica set cert-manager-webhook-5dbbbf75cc from 0 to 1

cert-manager

multus

cert-manager-webhook-5dbbbf75cc-zsl4r

AddedInterface

Add eth0 [10.133.0.19/23] from ovn-kubernetes

cert-manager

kubelet

cert-manager-webhook-5dbbbf75cc-zsl4r

Pulling

Pulling image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d"

opendatahub-operator-system

operator-lifecycle-manager

opendatahub-operator.v3.6.0-ea.2

InstallSucceeded

install strategy completed with no errors

cert-manager

kubelet

cert-manager-cainjector-5c57bbd4fc-x67vn

Pulled

Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 5.178s (5.178s including waiting). Image size: 309650901 bytes.

cert-manager

kubelet

cert-manager-cainjector-5c57bbd4fc-x67vn

Created

Container created

cert-manager

kubelet

cert-manager-cainjector-5c57bbd4fc-x67vn

Started

Container started
(x12)

cert-manager

replicaset-controller

cert-manager-597b95cb84

FailedCreate

Error creating: pods "cert-manager-597b95cb84-" is forbidden: error looking up service account cert-manager/cert-manager: serviceaccount "cert-manager" not found

cert-manager

kubelet

cert-manager-webhook-5dbbbf75cc-zsl4r

Pulled

Successfully pulled image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" in 2.944s (2.944s including waiting). Image size: 309650901 bytes.

cert-manager

kubelet

cert-manager-webhook-5dbbbf75cc-zsl4r

Created

Container created

cert-manager

kubelet

cert-manager-webhook-5dbbbf75cc-zsl4r

Started

Container started
(x2)

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

deployment-controller

kuadrant-operator-controller-manager

ScalingReplicaSet

Scaled up replica set kuadrant-operator-controller-manager-678d87766f from 0 to 1

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

InstallWaiting

installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

replicaset-controller

kuadrant-operator-controller-manager-678d87766f

SuccessfulCreate

Created pod: kuadrant-operator-controller-manager-678d87766f-4hmfk

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

RequirementsUnknown

requirements not yet checked

kuadrant-system

multus

kuadrant-operator-controller-manager-678d87766f-4hmfk

AddedInterface

Add eth0 [10.132.0.44/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-operator-controller-manager-678d87766f-4hmfk

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e"

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

replicaset-controller

authorino-operator-7b7bb4bc77

SuccessfulCreate

Created pod: authorino-operator-7b7bb4bc77-hshfp

kuadrant-system

deployment-controller

authorino-operator

ScalingReplicaSet

Scaled up replica set authorino-operator-7b7bb4bc77 from 0 to 1

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

RequirementsUnknown

requirements not yet checked

kuadrant-system

kubelet

authorino-operator-7b7bb4bc77-hshfp

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589"

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

InstallWaiting

installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.

kuadrant-system

multus

authorino-operator-7b7bb4bc77-hshfp

AddedInterface

Add eth0 [10.132.0.45/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

deployment-controller

limitador-operator-controller-manager

ScalingReplicaSet

Scaled up replica set limitador-operator-controller-manager-5778bb4449 from 0 to 1

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

replicaset-controller

limitador-operator-controller-manager-5778bb4449

SuccessfulCreate

Created pod: limitador-operator-controller-manager-5778bb4449-jtvzt

kuadrant-system

multus

limitador-operator-controller-manager-5778bb4449-jtvzt

AddedInterface

Add eth0 [10.133.0.20/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-operator-controller-manager-678d87766f-4hmfk

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e" in 6.259s (6.259s including waiting). Image size: 424244516 bytes.

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

InstallWaiting

installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

kubelet

kuadrant-operator-controller-manager-678d87766f-4hmfk

Created

Container created

kuadrant-system

kubelet

kuadrant-operator-controller-manager-678d87766f-4hmfk

Started

Container started

kuadrant-system

kubelet

limitador-operator-controller-manager-5778bb4449-jtvzt

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e"

kuadrant-system

kubelet

authorino-operator-7b7bb4bc77-hshfp

Created

Container created

kuadrant-system

kuadrant-operator-controller-manager-678d87766f-4hmfk_d3816548-27f4-4980-b7d0-dc620c706218

f139389e.kuadrant.io

LeaderElection

kuadrant-operator-controller-manager-678d87766f-4hmfk_d3816548-27f4-4980-b7d0-dc620c706218 became leader

kuadrant-system

authorino-operator-7b7bb4bc77-hshfp_7740ddf2-049b-40d6-9540-fc5755abe955

aac3a15d.authorino.kuadrant.io

LeaderElection

authorino-operator-7b7bb4bc77-hshfp_7740ddf2-049b-40d6-9540-fc5755abe955 became leader

kuadrant-system

kubelet

authorino-operator-7b7bb4bc77-hshfp

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589" in 3.155s (3.155s including waiting). Image size: 167738465 bytes.

kuadrant-system

kubelet

authorino-operator-7b7bb4bc77-hshfp

Started

Container started

kuadrant-system

kubelet

dns-operator-controller-manager-8557cbdb84-zttlw

Created

Container created

kuadrant-system

kubelet

dns-operator-controller-manager-8557cbdb84-zttlw

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a" in 15.801s (15.801s including waiting). Image size: 190804149 bytes.

cert-manager

replicaset-controller

cert-manager-597b95cb84

SuccessfulCreate

Created pod: cert-manager-597b95cb84-d2kl5

kuadrant-system

dns-operator-controller-manager-8557cbdb84-zttlw_923967fa-eb80-467b-b5f9-446bc240f874

a3f98d6c.kuadrant.io

LeaderElection

dns-operator-controller-manager-8557cbdb84-zttlw_923967fa-eb80-467b-b5f9-446bc240f874 became leader

kuadrant-system

kubelet

dns-operator-controller-manager-8557cbdb84-zttlw

Started

Container started

cert-manager

kubelet

cert-manager-597b95cb84-d2kl5

Started

Container started

cert-manager

kubelet

cert-manager-597b95cb84-d2kl5

Created

Container created

cert-manager

kubelet

cert-manager-597b95cb84-d2kl5

Pulled

Container image "registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:b3305a06bff4a344ba1b19c8142d7e605fc723841981c97aab2310ff8a82c66d" already present on machine and can be accessed by the pod

cert-manager

multus

cert-manager-597b95cb84-d2kl5

AddedInterface

Add eth0 [10.133.0.21/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-operator-controller-manager-5778bb4449-jtvzt

Created

Container created

kuadrant-system

deployment-controller

kuadrant-console-plugin

ScalingReplicaSet

Scaled up replica set kuadrant-console-plugin-66c4fb575d from 0 to 1

kuadrant-system

kubelet

limitador-operator-controller-manager-5778bb4449-jtvzt

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e" in 4.044s (4.044s including waiting). Image size: 177967631 bytes.

kuadrant-system

replicaset-controller

kuadrant-console-plugin-66c4fb575d

SuccessfulCreate

Created pod: kuadrant-console-plugin-66c4fb575d-hb9vr

kuadrant-system

kubelet

limitador-operator-controller-manager-5778bb4449-jtvzt

Started

Container started

kuadrant-system

multus

kuadrant-console-plugin-66c4fb575d-hb9vr

AddedInterface

Add eth0 [10.134.0.25/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-console-plugin-66c4fb575d-hb9vr

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044"

kuadrant-system

limitador-operator-controller-manager-5778bb4449-jtvzt_ed82dbe7-714a-4a65-a68a-a709fd206892

3745a16e.kuadrant.io

LeaderElection

limitador-operator-controller-manager-5778bb4449-jtvzt_ed82dbe7-714a-4a65-a68a-a709fd206892 became leader

kuadrant-system

kubelet

kuadrant-console-plugin-66c4fb575d-hb9vr

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:c8550de6790d58c5fa993bf83d41ef693d99a5739fcc0eed3fcaa8d5a7658044" in 5.943s (5.943s including waiting). Image size: 342538866 bytes.

kuadrant-system

kubelet

kuadrant-console-plugin-66c4fb575d-hb9vr

Created

Container created

kuadrant-system

kubelet

kuadrant-console-plugin-66c4fb575d-hb9vr

Started

Container started

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.3

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.3

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.2

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.2

InstallSucceeded

install strategy completed with no errors

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleCreated

Created ClusterRole.rbac.authorization.k8s.io/jobset-metrics-reader because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleBindingCreated

Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-metrics-reader-rolebinding because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

ServiceCreated

Created Service/jobset-webhook-service -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleCreated

Created ClusterRole.rbac.authorization.k8s.io/jobset-manager-role because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleCreated

Created ClusterRole.rbac.authorization.k8s.io/jobset-proxy-role because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ServiceAccountCreated

Created ServiceAccount/jobset-controller-manager -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

IssuerCreated

Created Issuer.cert-manager.io/jobset-selfsigned-issuer -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleBindingCreated

Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-manager-rolebinding because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

CertificateCreated

Created Certificate.cert-manager.io/jobset-serving-cert -n openshift-jobset-operator because it was missing

openshift-jobset-operator

cert-manager-certificaterequests-issuer-selfsigned

jobset-metrics-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

RoleCreated

Created Role.rbac.authorization.k8s.io/jobset-leader-election-role -n openshift-jobset-operator because it was missing

openshift-jobset-operator

cert-manager-certificaterequests-issuer-ca

jobset-metrics-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificaterequests-issuer-vault

jobset-metrics-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificaterequests-issuer-venafi

jobset-metrics-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ClusterRoleBindingCreated

Created ClusterRoleBinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

CertificateCreated

Created Certificate.cert-manager.io/jobset-metrics-cert -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

RoleCreated

Created Role.rbac.authorization.k8s.io/jobset-manager-secrets-role -n openshift-jobset-operator because it was missing

openshift-jobset-operator

cert-manager-certificaterequests-issuer-acme

jobset-metrics-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificates-request-manager

jobset-serving-cert

Requested

Created new CertificateRequest resource "jobset-serving-cert-1"

openshift-jobset-operator

cert-manager-certificates-key-manager

jobset-serving-cert

Generated

Stored new private key in temporary Secret resource "jobset-serving-cert-l4fcx"

openshift-jobset-operator

cert-manager-certificates-trigger

jobset-serving-cert

Issuing

Issuing certificate as Secret does not exist

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

RoleBindingCreated

Created RoleBinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding -n openshift-jobset-operator because it was missing

openshift-jobset-operator

cert-manager-certificates-trigger

jobset-metrics-cert

Issuing

Issuing certificate as Secret does not exist

openshift-jobset-operator

cert-manager-certificaterequests-issuer-ca

jobset-serving-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificates-request-manager

jobset-metrics-cert

Requested

Created new CertificateRequest resource "jobset-metrics-cert-1"

openshift-jobset-operator

cert-manager-certificaterequests-issuer-selfsigned

jobset-serving-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificaterequests-issuer-vault

jobset-serving-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificaterequests-issuer-acme

jobset-serving-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificaterequests-issuer-venafi

jobset-serving-cert-1

WaitingForApproval

Not signing CertificateRequest until it is Approved

openshift-jobset-operator

cert-manager-certificates-key-manager

jobset-metrics-cert

Generated

Stored new private key in temporary Secret resource "jobset-metrics-cert-k8zzj"

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

RoleBindingCreated

Created RoleBinding.rbac.authorization.k8s.io/jobset-manager-secrets-rolebinding -n openshift-jobset-operator because it was missing

openshift-ingress

replicaset-controller

kube-auth-proxy-7659c7d778

SuccessfulCreate

Created pod: kube-auth-proxy-7659c7d778-65h9l

openshift-ingress

deployment-controller

data-science-gateway-data-science-gateway-class

ScalingReplicaSet

Scaled up replica set data-science-gateway-data-science-gateway-class-67b5c995fc from 0 to 1

openshift-jobset-operator

cert-manager-certificaterequests-approver

jobset-serving-cert-1

cert-manager.io

Certificate request has been approved by cert-manager.io

openshift-jobset-operator

cert-manager-certificaterequests-approver

jobset-metrics-cert-1

cert-manager.io

Certificate request has been approved by cert-manager.io

openshift-ingress

multus

kube-auth-proxy-7659c7d778-lxgpd

AddedInterface

Add eth0 [10.133.0.22/23] from ovn-kubernetes

openshift-jobset-operator

cert-manager-certificaterequests-issuer-selfsigned

jobset-metrics-cert-1

CertificateIssued

Certificate fetched from issuer successfully

openshift-ingress

replicaset-controller

data-science-gateway-data-science-gateway-class-67b5c995fc

SuccessfulCreate

Created pod: data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

openshift-ingress

deployment-controller

kube-auth-proxy

ScalingReplicaSet

Scaled up replica set kube-auth-proxy-7659c7d778 from 0 to 2

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-lxgpd

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"

openshift-ingress

replicaset-controller

kube-auth-proxy-7659c7d778

SuccessfulCreate

Created pod: kube-auth-proxy-7659c7d778-lxgpd

openshift-ingress

multus

kube-auth-proxy-7659c7d778-65h9l

AddedInterface

Add eth0 [10.134.0.26/23] from ovn-kubernetes

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

ValidatingWebhookConfigurationCreated

Created ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it was missing

openshift-jobset-operator

cert-manager-certificates-issuing

jobset-metrics-cert

Issuing

The certificate has been successfully issued

openshift-jobset-operator

cert-manager-certificates-issuing

jobset-serving-cert

Issuing

The certificate has been successfully issued

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

MutatingWebhookConfigurationCreated

Created MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it was missing

openshift-jobset-operator

cert-manager-certificaterequests-issuer-selfsigned

jobset-serving-cert-1

CertificateIssued

Certificate fetched from issuer successfully

openshift-jobset-operator

openshift-jobset-operator-jobsetoperatorstaticresources-jobsetoperatorstaticresources-staticresources

jobset-operator

ServiceCreated

Created Service/jobset-controller-manager-metrics-service -n openshift-jobset-operator because it was missing

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

Created

Container created

openshift-ingress

multus

data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

AddedInterface

Add eth0 [10.132.0.46/23] from ovn-kubernetes

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

CustomResourceDefinitionCreated

Created CustomResourceDefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io because it was missing

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

Started

Container started

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-65h9l

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035"

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-67b5c995fcd6p9c

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-74dfcfb894 from 0 to 1

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

RoleCreated

Created Role.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing

kuadrant-system

replicaset-controller

limitador-limitador-74dfcfb894

SuccessfulCreate

Created pod: limitador-limitador-74dfcfb894-hvj2g

kuadrant-system

multus

limitador-limitador-74dfcfb894-hvj2g

AddedInterface

Add eth0 [10.132.0.47/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-limitador-74dfcfb894-hvj2g

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

RoleBindingCreated

Created RoleBinding.rbac.authorization.k8s.io/jobset-prometheus-k8s -n openshift-jobset-operator because it was missing

openshift-jobset-operator

deployment-controller

jobset-controller-manager

ScalingReplicaSet

Scaled up replica set jobset-controller-manager-696bfff7cc from 0 to 1

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

ServiceMonitorCreated

Created ServiceMonitor.monitoring.coreos.com/jobset-controller-manager-metrics-monitor -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

ConfigMapCreated

Created ConfigMap/jobset-manager-config -n openshift-jobset-operator because it was missing

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

DeploymentCreated

Created Deployment.apps/jobset-controller-manager -n openshift-jobset-operator because it was missing

openshift-jobset-operator

replicaset-controller

jobset-controller-manager-696bfff7cc

SuccessfulCreate

Created pod: jobset-controller-manager-696bfff7cc-5rxhf

kuadrant-system

replicaset-controller

authorino-57f7f8659f

SuccessfulCreate

Created pod: authorino-57f7f8659f-mqhqz

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-57f7f8659f from 0 to 1

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

ValidatingWebhookConfigurationUpdated

Updated ValidatingWebhookConfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration because it changed

openshift-jobset-operator

openshift-jobset-operator

jobset-operator

MutatingWebhookConfigurationUpdated

Updated MutatingWebhookConfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration because it changed

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-lxgpd

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.532s (3.532s including waiting). Image size: 188904677 bytes.

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-lxgpd

Started

Container started

kuadrant-system

multus

authorino-57f7f8659f-mqhqz

AddedInterface

Add eth0 [10.132.0.48/23] from ovn-kubernetes

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-65h9l

Started

Container started

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-65h9l

Created

Container created

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-65h9l

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.399s (3.399s including waiting). Image size: 188904677 bytes.

kuadrant-system

kubelet

authorino-57f7f8659f-mqhqz

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

openshift-ingress

kubelet

kube-auth-proxy-7659c7d778-lxgpd

Created

Container created

openshift-jobset-operator

multus

jobset-controller-manager-696bfff7cc-5rxhf

AddedInterface

Add eth0 [10.134.0.27/23] from ovn-kubernetes

openshift-jobset-operator

kubelet

jobset-controller-manager-696bfff7cc-5rxhf

Pulling

Pulling image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987"

kuadrant-system

kubelet

limitador-limitador-74dfcfb894-hvj2g

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 2.462s (2.462s including waiting). Image size: 133379307 bytes.

kuadrant-system

replicaset-controller

authorino-56c6c9ddd5

SuccessfulCreate

Created pod: authorino-56c6c9ddd5-2f296

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-7c9b87d574 from 0 to 1

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-56c6c9ddd5 from 0 to 1

kuadrant-system

replicaset-controller

limitador-limitador-7c9b87d574

SuccessfulCreate

Created pod: limitador-limitador-7c9b87d574-rqz2j

kuadrant-system

kubelet

limitador-limitador-7c9b87d574-rqz2j

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751"

kuadrant-system

multus

authorino-56c6c9ddd5-2f296

AddedInterface

Add eth0 [10.133.0.23/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-56c6c9ddd5-2f296

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

kuadrant-system

kubelet

limitador-limitador-74dfcfb894-hvj2g

Created

Container created

kuadrant-system

kubelet

limitador-limitador-74dfcfb894-hvj2g

Started

Container started

kuadrant-system

multus

limitador-limitador-7c9b87d574-rqz2j

AddedInterface

Add eth0 [10.133.0.24/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-57f7f8659f-mqhqz

Created

Container created

openshift-jobset-operator

kubelet

jobset-controller-manager-696bfff7cc-5rxhf

Created

Container created

kuadrant-system

kubelet

authorino-57f7f8659f-mqhqz

Started

Container started

openshift-jobset-operator

kubelet

jobset-controller-manager-696bfff7cc-5rxhf

Started

Container started

openshift-jobset-operator

kubelet

jobset-controller-manager-696bfff7cc-5rxhf

Pulled

Successfully pulled image "registry.redhat.io/job-set/jobset-rhel9@sha256:8d2a70fe90b2414cdb269d217703b7e5b3409f9276acf671721b1abc67a0b987" in 4.292s (4.292s including waiting). Image size: 182184678 bytes.

kuadrant-system

kubelet

authorino-57f7f8659f-mqhqz

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 3.39s (3.39s including waiting). Image size: 204096810 bytes.

openshift-jobset-operator

jobset-controller-manager-696bfff7cc-5rxhf_10c2cfe8-2a86-4973-ae36-4944abbb914d

6d4f6a47.jobset.x-k8s.io

LeaderElection

jobset-controller-manager-696bfff7cc-5rxhf_10c2cfe8-2a86-4973-ae36-4944abbb914d became leader

kuadrant-system

kubelet

authorino-56c6c9ddd5-2f296

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 2.661s (2.661s including waiting). Image size: 204096810 bytes.

kuadrant-system

kubelet

authorino-56c6c9ddd5-2f296

Created

Container created

kuadrant-system

kubelet

authorino-56c6c9ddd5-2f296

Started

Container started

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-57f7f8659f from 1 to 0

kuadrant-system

replicaset-controller

authorino-57f7f8659f

SuccessfulDelete

Deleted pod: authorino-57f7f8659f-mqhqz

kuadrant-system

kubelet

authorino-57f7f8659f-mqhqz

Killing

Stopping container authorino

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

data-science-gateway-data-science-gateway-class

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

data-science-gateway-data-science-gateway-class

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kuadrant-system

kubelet

limitador-limitador-7c9b87d574-rqz2j

Created

Container created

kuadrant-system

kubelet

limitador-limitador-7c9b87d574-rqz2j

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:793a38ce1b5690b99bc20fb7fca3304727d5fb21fcf440ab88f3e7af5778a751" in 12.653s (12.653s including waiting). Image size: 133379307 bytes.

kuadrant-system

kubelet

limitador-limitador-7c9b87d574-rqz2j

Started

Container started

opendatahub

deployment-controller

kserve-module-controller-manager

ScalingReplicaSet

Scaled up replica set kserve-module-controller-manager-7566cf5fc8 from 0 to 1

opendatahub

replicaset-controller

kserve-module-controller-manager-7566cf5fc8

SuccessfulCreate

Created pod: kserve-module-controller-manager-7566cf5fc8-qhqnk

opendatahub

multus

kserve-module-controller-manager-7566cf5fc8-qhqnk

AddedInterface

Add eth0 [10.132.0.49/23] from ovn-kubernetes

opendatahub

kubelet

kserve-module-controller-manager-7566cf5fc8-qhqnk

Pulling

Pulling image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189"

opendatahub

kubelet

kserve-module-controller-manager-7566cf5fc8-qhqnk

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kserve-module-operator@sha256:b8becaa017a2a2ec9af002989e8f14a8a2f96ffad292f37a61569343b1101189" in 2.922s (2.922s including waiting). Image size: 195729000 bytes.

opendatahub

kubelet

kserve-module-controller-manager-7566cf5fc8-qhqnk

Started

Container started

opendatahub

kubelet

kserve-module-controller-manager-7566cf5fc8-qhqnk

Created

Container created

kuadrant-system

kubelet

limitador-limitador-74dfcfb894-hvj2g

Killing

Stopping container limitador

kuadrant-system

replicaset-controller

limitador-limitador-74dfcfb894

SuccessfulDelete

Deleted pod: limitador-limitador-74dfcfb894-hvj2g

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled down replica set limitador-limitador-74dfcfb894 from 1 to 0

openshift-ingress

horizontal-pod-autoscaler

data-science-gateway-data-science-gateway-class

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

data-science-gateway-data-science-gateway-class

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

opendatahub

kserve-module-controller-manager-7566cf5fc8-qhqnk_5593ad0e-d77e-4e94-9e38-b063236a09ad

kserve-module-controller-manager

LeaderElection

kserve-module-controller-manager-7566cf5fc8-qhqnk_5593ad0e-d77e-4e94-9e38-b063236a09ad became leader

opendatahub

replicaset-controller

maas-controller-589f8697fb

SuccessfulCreate

Created pod: maas-controller-589f8697fb-5r6hk

opendatahub

deployment-controller

maas-controller

ScalingReplicaSet

Scaled up replica set maas-controller-589f8697fb from 0 to 1

opendatahub

replicaset-controller

maas-controller-5fb896bc89

SuccessfulCreate

Created pod: maas-controller-5fb896bc89-fkk4g

opendatahub

multus

maas-controller-5fb896bc89-fkk4g

AddedInterface

Add eth0 [10.132.0.52/23] from ovn-kubernetes

opendatahub

multus

maas-controller-589f8697fb-5r6hk

AddedInterface

Add eth0 [10.132.0.51/23] from ovn-kubernetes

opendatahub

kubelet

maas-controller-589f8697fb-5r6hk

Pulling

Pulling image "quay.io/opendatahub/maas-controller@sha256:6427f69265e0a63a4419ff762ef9889590a3f880d23acaddb56f5d9a5c08b665"

opendatahub

deployment-controller

maas-controller

ScalingReplicaSet

Scaled up replica set maas-controller-5fb896bc89 from 0 to 1

opendatahub

kubelet

maas-controller-589f8697fb-5r6hk

Started

Container started

opendatahub

kubelet

maas-controller-589f8697fb-5r6hk

Pulled

Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:6427f69265e0a63a4419ff762ef9889590a3f880d23acaddb56f5d9a5c08b665" in 6.979s (6.979s including waiting). Image size: 200111630 bytes.

opendatahub

kubelet

maas-controller-589f8697fb-5r6hk

Created

Container created

opendatahub

kubelet

maas-controller-5fb896bc89-fkk4g

Pulled

Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:6427f69265e0a63a4419ff762ef9889590a3f880d23acaddb56f5d9a5c08b665" in 6.795s (6.795s including waiting). Image size: 200111630 bytes.

opendatahub

maas-controller-589f8697fb-5r6hk_bfaaa753-442d-4662-a2d7-2c3cd0723575

maas-controller.models-as-a-service.opendatahub.io

LeaderElection

maas-controller-589f8697fb-5r6hk_bfaaa753-442d-4662-a2d7-2c3cd0723575 became leader

opendatahub

deployment-controller

usage-logs-tenancy-proxy

ScalingReplicaSet

Scaled up replica set usage-logs-tenancy-proxy-9c5d49bcd from 0 to 1

opendatahub

replicaset-controller

usage-logs-tenancy-proxy-9c5d49bcd

SuccessfulCreate

Created pod: usage-logs-tenancy-proxy-9c5d49bcd-98hr6

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

FailedMount

MountVolume.SetUp failed for volume "proxy-source" : configmap "usage-tenancy-proxy-source" not found

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

FailedMount

MountVolume.SetUp failed for volume "tls-cert" : secret "usage-logs-tenancy-proxy-tls" not found

opendatahub

multus

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

AddedInterface

Add eth0 [10.132.0.53/23] from ovn-kubernetes

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

Pulling

Pulling image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172"

opendatahub

replicaset-controller

maas-controller-589f8697fb

SuccessfulDelete

Deleted pod: maas-controller-589f8697fb-5r6hk

opendatahub

deployment-controller

maas-controller

ScalingReplicaSet

Scaled down replica set maas-controller-589f8697fb from 1 to 0

opendatahub

kubelet

maas-controller-589f8697fb-5r6hk

Killing

Stopping container manager

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7ddd9cfdc8

SuccessfulCreate

Created pod: istiod-openshift-gateway-7ddd9cfdc8-pmscg

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

Pulled

Successfully pulled image "registry.redhat.io/ubi9/python-312@sha256:f6713d327d37e654a443752e6654b5aab88f31690e1161eed9c34dd837870172" in 12.669s (12.669s including waiting). Image size: 1089502687 bytes.

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

Created

Container created

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8"

openshift-ingress

multus

istiod-openshift-gateway-7ddd9cfdc8-pmscg

AddedInterface

Add eth0 [10.132.0.54/23] from ovn-kubernetes

opendatahub

kubelet

usage-logs-tenancy-proxy-9c5d49bcd-98hr6

Started

Container started

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Started

Container started

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Created

Container created

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.116s (4.116s including waiting). Image size: 185476477 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

opendatahub

replicaset-controller

kserve-controller-manager-776cc66d47

SuccessfulCreate

Created pod: kserve-controller-manager-776cc66d47-rcb44

opendatahub

deployment-controller

model-serving-api

ScalingReplicaSet

Scaled up replica set model-serving-api-75bd89fdfc from 0 to 1

opendatahub

deployment-controller

kserve-controller-manager

ScalingReplicaSet

Scaled up replica set kserve-controller-manager-776cc66d47 from 0 to 1

opendatahub

replicaset-controller

model-serving-api-75bd89fdfc

SuccessfulCreate

Created pod: model-serving-api-75bd89fdfc-tkm8m

opendatahub

multus

kserve-controller-manager-776cc66d47-rcb44

AddedInterface

Add eth0 [10.133.0.25/23] from ovn-kubernetes

opendatahub

deployment-controller

llmisvc-controller-manager

ScalingReplicaSet

Scaled up replica set llmisvc-controller-manager-869d5dcc5d from 0 to 1

opendatahub

replicaset-controller

llmisvc-controller-manager-869d5dcc5d

SuccessfulCreate

Created pod: llmisvc-controller-manager-869d5dcc5d-w452v

openshift-ingress

kubelet

istiod-openshift-gateway-7ddd9cfdc8-pmscg

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

opendatahub

replicaset-controller

odh-model-controller-769875db56

SuccessfulCreate

Created pod: odh-model-controller-769875db56-n2wm8

opendatahub

deployment-controller

odh-model-controller

ScalingReplicaSet

Scaled up replica set odh-model-controller-769875db56 from 0 to 1

opendatahub

kubelet

kserve-controller-manager-776cc66d47-rcb44

Pulling

Pulling image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d"

opendatahub

kubelet

llmisvc-controller-manager-869d5dcc5d-w452v

Pulling

Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"

opendatahub

multus

llmisvc-controller-manager-869d5dcc5d-w452v

AddedInterface

Add eth0 [10.133.0.26/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7876bfcb74

SuccessfulDelete

Deleted pod: istiod-openshift-gateway-7876bfcb74-mzwml

opendatahub

kubelet

odh-model-controller-769875db56-n2wm8

Pulling

Pulling image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775"

opendatahub

multus

odh-model-controller-769875db56-n2wm8

AddedInterface

Add eth0 [10.134.0.28/23] from ovn-kubernetes

opendatahub

multus

model-serving-api-75bd89fdfc-tkm8m

AddedInterface

Add eth0 [10.133.0.27/23] from ovn-kubernetes

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0

openshift-ingress

kubelet

istiod-openshift-gateway-7876bfcb74-mzwml

Killing

Stopping container discovery

opendatahub

kubelet

model-serving-api-75bd89fdfc-tkm8m

Pulling

Pulling image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8"

opendatahub

kubelet

model-serving-api-75bd89fdfc-tkm8m

Pulled

Successfully pulled image "quay.io/opendatahub/odh-model-serving-api@sha256:dc6ced7ac752d3770e31e3e83f4f9a6c793de5b9b04dde0d194d021020cef8e8" in 2.547s (2.547s including waiting). Image size: 184925761 bytes.

opendatahub

kubelet

model-serving-api-75bd89fdfc-tkm8m

Created

Container created

opendatahub

kubelet

model-serving-api-75bd89fdfc-tkm8m

Started

Container started

opendatahub

maas-controller-5fb896bc89-fkk4g_039a2f0d-18ef-41c2-83d4-ccc513d55556

maas-controller.models-as-a-service.opendatahub.io

LeaderElection

maas-controller-5fb896bc89-fkk4g_039a2f0d-18ef-41c2-83d4-ccc513d55556 became leader

opendatahub

kubelet

kserve-controller-manager-776cc66d47-rcb44

Created

Container created

opendatahub

kubelet

kserve-controller-manager-776cc66d47-rcb44

Started

Container started

opendatahub

kubelet

odh-model-controller-769875db56-n2wm8

Created

Container created

opendatahub

kubelet

llmisvc-controller-manager-869d5dcc5d-w452v

Started

Container started

opendatahub

kubelet

odh-model-controller-769875db56-n2wm8

Started

Container started

opendatahub

kubelet

odh-model-controller-769875db56-n2wm8

Pulled

Successfully pulled image "quay.io/opendatahub/odh-model-controller@sha256:94f7f4383ef91d10d60d82c0858a60158a02915fdd3000d507210139faaac775" in 3.184s (3.184s including waiting). Image size: 236755567 bytes.

opendatahub

kubelet

kserve-controller-manager-776cc66d47-rcb44

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d" in 3.273s (3.273s including waiting). Image size: 247676581 bytes.

opendatahub

kubelet

llmisvc-controller-manager-869d5dcc5d-w452v

Created

Container created

opendatahub

kubelet

llmisvc-controller-manager-869d5dcc5d-w452v

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 3.014s (3.014s including waiting). Image size: 235100981 bytes.

opendatahub

odh-model-controller-769875db56-n2wm8_7f8fe5f1-b978-4c12-8837-1bce5e8b4da1

odh-model-controller.opendatahub.io

LeaderElection

odh-model-controller-769875db56-n2wm8_7f8fe5f1-b978-4c12-8837-1bce5e8b4da1 became leader

opendatahub

kserve-controller-manager-776cc66d47-rcb44_dd9fe941-0369-424d-80ed-c65d9785e6ae

kserve-controller-manager-leader-lock

LeaderElection

kserve-controller-manager-776cc66d47-rcb44_dd9fe941-0369-424d-80ed-c65d9785e6ae became leader

opendatahub

llmisvc-controller-manager-869d5dcc5d-w452v_ee97686c-5f3a-414d-bf12-995a8964eacc

llminferenceservice-kserve-controller-manager

LeaderElection

llmisvc-controller-manager-869d5dcc5d-w452v_ee97686c-5f3a-414d-bf12-995a8964eacc became leader

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

ProbeError

Readiness probe error: Get "http://10.132.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Unhealthy

Readiness probe failed: Get "http://10.132.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Created

Container created
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-6f6944bcdd from 0 to 1

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kuadrant-system

replicaset-controller

authorino-6f6944bcdd

SuccessfulCreate

Created pod: authorino-6f6944bcdd-9dzz8

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-76f555cd65 from 0 to 1

kuadrant-system

multus

authorino-76f555cd65-p5s8m

AddedInterface

Add eth0 [10.133.0.29/23] from ovn-kubernetes

kuadrant-system

replicaset-controller

authorino-6f6944bcdd

SuccessfulDelete

Deleted pod: authorino-6f6944bcdd-9dzz8

kuadrant-system

replicaset-controller

authorino-6f7c4f5d87

SuccessfulCreate

Created pod: authorino-6f7c4f5d87-28j47

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-6f7c4f5d87 from 0 to 1

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-76f555cd65 from 1 to 0

kuadrant-system

multus

authorino-6f6944bcdd-9dzz8

AddedInterface

Add eth0 [10.133.0.28/23] from ovn-kubernetes

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-6f6944bcdd from 1 to 0

kuadrant-system

replicaset-controller

authorino-76f555cd65

SuccessfulCreate

Created pod: authorino-76f555cd65-p5s8m

kuadrant-system

kubelet

authorino-6f6944bcdd-9dzz8

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

kuadrant-system

replicaset-controller

authorino-76f555cd65

SuccessfulDelete

Deleted pod: authorino-76f555cd65-p5s8m

kuadrant-system

kubelet

authorino-6f7c4f5d87-28j47

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

kuadrant-system

multus

authorino-6f7c4f5d87-28j47

AddedInterface

Add eth0 [10.132.0.55/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-76f555cd65-p5s8m

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

kuadrant-system

kubelet

authorino-6f6944bcdd-9dzz8

Started

Container started

kuadrant-system

kubelet

authorino-6f7c4f5d87-28j47

Started

Container started

kuadrant-system

kubelet

authorino-76f555cd65-p5s8m

Created

Container created

kuadrant-system

kubelet

authorino-6f6944bcdd-9dzz8

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.785s (1.785s including waiting). Image size: 204096810 bytes.

kuadrant-system

kubelet

authorino-6f6944bcdd-9dzz8

Created

Container created

kuadrant-system

kubelet

authorino-6f7c4f5d87-28j47

Created

Container created

kuadrant-system

kubelet

authorino-76f555cd65-p5s8m

Started

Container started

kuadrant-system

kubelet

authorino-6f7c4f5d87-28j47

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.421s (1.421s including waiting). Image size: 204096810 bytes.

kuadrant-system

kubelet

authorino-76f555cd65-p5s8m

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.524s (1.524s including waiting). Image size: 204096810 bytes.

kuadrant-system

replicaset-controller

authorino-56c6c9ddd5

SuccessfulDelete

Deleted pod: authorino-56c6c9ddd5-2f296

kuadrant-system

kubelet

authorino-6f6944bcdd-9dzz8

Killing

Stopping container authorino

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-56c6c9ddd5 from 1 to 0
(x5)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

Unhealthy

Startup probe failed: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-v5h77

ProbeError

Startup probe error: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused body:

kuadrant-system

kubelet

authorino-76f555cd65-p5s8m

Killing

Stopping container authorino

kuadrant-system

kubelet

authorino-56c6c9ddd5-2f296

Killing

Stopping container authorino
(x5)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

opendatahub

multus

llmisvc-controller-manager-6466c554d8-fbbwk

AddedInterface

Add eth0 [10.133.0.30/23] from ovn-kubernetes

opendatahub

deployment-controller

llmisvc-controller-manager

ScalingReplicaSet

Scaled up replica set llmisvc-controller-manager-6466c554d8 from 0 to 1

opendatahub

kubelet

llmisvc-controller-manager-6466c554d8-fbbwk

Pulling

Pulling image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321"

opendatahub

replicaset-controller

llmisvc-controller-manager-6466c554d8

SuccessfulCreate

Created pod: llmisvc-controller-manager-6466c554d8-fbbwk

opendatahub

kubelet

llmisvc-controller-manager-6466c554d8-fbbwk

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:be3760deb2e1570f182db0819a98b4cebd1de04de25b8c9bdd0b38396f0be321" in 897ms (897ms including waiting). Image size: 235100981 bytes.

opendatahub

kubelet

llmisvc-controller-manager-6466c554d8-fbbwk

Created

Container created

opendatahub

kubelet

llmisvc-controller-manager-6466c554d8-fbbwk

Started

Container started

opendatahub

kubelet

llmisvc-controller-manager-869d5dcc5d-w452v

Killing

Stopping container manager

opendatahub

replicaset-controller

llmisvc-controller-manager-869d5dcc5d

SuccessfulDelete

Deleted pod: llmisvc-controller-manager-869d5dcc5d-w452v

opendatahub

deployment-controller

llmisvc-controller-manager

ScalingReplicaSet

Scaled down replica set llmisvc-controller-manager-869d5dcc5d from 1 to 0

opendatahub

deployment-controller

ai-gateway-controller

ScalingReplicaSet

Scaled up replica set ai-gateway-controller-5d8dddb676 from 0 to 1

opendatahub

replicaset-controller

ai-gateway-controller-5d8dddb676

SuccessfulCreate

Created pod: ai-gateway-controller-5d8dddb676-jcvfc

opendatahub

multus

ai-gateway-controller-5d8dddb676-jcvfc

AddedInterface

Add eth0 [10.134.0.31/23] from ovn-kubernetes

opendatahub

kubelet

ai-gateway-controller-5d8dddb676-jcvfc

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"

opendatahub

kubelet

ai-gateway-controller-5d8dddb676-jcvfc

Started

Container started

opendatahub

kubelet

ai-gateway-controller-5d8dddb676-jcvfc

Created

Container created

opendatahub

kubelet

ai-gateway-controller-5d8dddb676-jcvfc

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 2.222s (2.222s including waiting). Image size: 184136103 bytes.

opendatahub

ai-gateway-controller-5d8dddb676-jcvfc_b06add37-fafd-456f-b0f1-5f6d5b0a3700

ai-gateway-controller-leader.opendatahub.io

LeaderElection

ai-gateway-controller-5d8dddb676-jcvfc_b06add37-fafd-456f-b0f1-5f6d5b0a3700 became leader

openshift-ingress

replicaset-controller

payload-processing-7bb9489676

SuccessfulCreate

Created pod: payload-processing-7bb9489676-khhp6

openshift-ingress

deployment-controller

payload-processing

ScalingReplicaSet

Scaled up replica set payload-processing-7bb9489676 from 0 to 1
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-7b98bb678c

FailedCreate

Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found

openshift-ingress

replicaset-controller

payload-pre-processing-7b98bb678c

SuccessfulCreate

Created pod: payload-pre-processing-7b98bb678c-chld2

openshift-ingress

deployment-controller

payload-pre-processing

ScalingReplicaSet

Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1
(x7)

openshift-ingress

replicaset-controller

payload-processing-7bb9489676

FailedCreate

Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found

openshift-ingress

multus

payload-pre-processing-7b98bb678c-chld2

AddedInterface

Add eth0 [10.134.0.32/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-7b98bb678c-chld2

Pulling

Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"

openshift-ingress

multus

payload-processing-7bb9489676-khhp6

AddedInterface

Add eth0 [10.134.0.33/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-7bb9489676-khhp6

Pulling

Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"

openshift-ingress

kubelet

payload-processing-7bb9489676-khhp6

Pulled

Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.97s (2.97s including waiting). Image size: 125353963 bytes.

openshift-ingress

kubelet

payload-pre-processing-7b98bb678c-chld2

Created

Container created

openshift-ingress

kubelet

payload-processing-7bb9489676-khhp6

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-7b98bb678c-chld2

Pulled

Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.075s (3.075s including waiting). Image size: 125353963 bytes.

openshift-ingress

kubelet

payload-pre-processing-7b98bb678c-chld2

Started

Container started

openshift-ingress

kubelet

payload-processing-7bb9489676-khhp6

Created

Container created

opendatahub

replicaset-controller

ai-gateway-controller-7cd9b8cccb

SuccessfulCreate

Created pod: ai-gateway-controller-7cd9b8cccb-7bml5

opendatahub

deployment-controller

ai-gateway-controller

ScalingReplicaSet

Scaled up replica set ai-gateway-controller-7cd9b8cccb from 0 to 1

opendatahub

llmisvc-controller-manager-6466c554d8-fbbwk_aec39b1a-72ed-49c3-ab17-0a249615c518

llminferenceservice-kserve-controller-manager

LeaderElection

llmisvc-controller-manager-6466c554d8-fbbwk_aec39b1a-72ed-49c3-ab17-0a249615c518 became leader

opendatahub

multus

ai-gateway-controller-7cd9b8cccb-7bml5

AddedInterface

Add eth0 [10.134.0.34/23] from ovn-kubernetes

opendatahub

kubelet

ai-gateway-controller-7cd9b8cccb-7bml5

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-controller:latest"

opendatahub

kubelet

ai-gateway-controller-7cd9b8cccb-7bml5

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-controller:latest" in 905ms (905ms including waiting). Image size: 184136103 bytes.

opendatahub

kubelet

ai-gateway-controller-7cd9b8cccb-7bml5

Started

Container started

opendatahub

kubelet

ai-gateway-controller-7cd9b8cccb-7bml5

Created

Container created
(x2)

opendatahub

kubelet

maas-controller-5fb896bc89-fkk4g

Pulling

Pulling image "quay.io/opendatahub/maas-controller@sha256:6427f69265e0a63a4419ff762ef9889590a3f880d23acaddb56f5d9a5c08b665"
(x2)

opendatahub

kubelet

maas-controller-5fb896bc89-fkk4g

Created

Container created
(x2)

opendatahub

kubelet

maas-controller-5fb896bc89-fkk4g

Started

Container started

opendatahub

kubelet

maas-controller-5fb896bc89-fkk4g

Pulled

Successfully pulled image "quay.io/opendatahub/maas-controller@sha256:6427f69265e0a63a4419ff762ef9889590a3f880d23acaddb56f5d9a5c08b665" in 1.254s (1.254s including waiting). Image size: 200111630 bytes.

opendatahub

kubelet

ai-gateway-controller-5d8dddb676-jcvfc

Killing

Stopping container manager

opendatahub

deployment-controller

ai-gateway-controller

ScalingReplicaSet

Scaled down replica set ai-gateway-controller-5d8dddb676 from 1 to 0

opendatahub

replicaset-controller

ai-gateway-controller-5d8dddb676

SuccessfulDelete

Deleted pod: ai-gateway-controller-5d8dddb676-jcvfc

opendatahub

maas-controller-5fb896bc89-fkk4g_3abc8a56-1004-458b-9a1b-cce10d8ff9ab

maas-controller.models-as-a-service.opendatahub.io

LeaderElection

maas-controller-5fb896bc89-fkk4g_3abc8a56-1004-458b-9a1b-cce10d8ff9ab became leader

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.Service llm/e2e-distinct-2-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.Deployment llm/e2e-distinct-2-simulated-kserve

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs

llm

deployment-controller

e2e-distinct-2-simulated-kserve

ScalingReplicaSet

Scaled up replica set e2e-distinct-2-simulated-kserve-cbd4f5586 from 0 to 1

llm

replicaset-controller

e2e-distinct-2-simulated-kserve-cbd4f5586

SuccessfulCreate

Created pod: e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.HTTPRoute llm/e2e-distinct-2-simulated-kserve-route

llm

multus

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

AddedInterface

Add eth0 [10.134.0.35/23] from ovn-kubernetes

llm

kubelet

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.DestinationRule llm/e2e-distinct-2-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.PodMonitor llm/e2e-distinct-2-simulated-kserve-llmisvc-engine

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Created

Created v1.NetworkPolicy llm/e2e-distinct-2-simulated-prometheus-scraping

llm

kubelet

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

Started

Container started

llm

kubelet

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.053s (2.053s including waiting). Image size: 98346788 bytes.

llm

kubelet

e2e-distinct-2-simulated-kserve-cbd4f5586-dr574

Created

Container created

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.DestinationRule llm/e2e-distinct-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.HTTPRoute llm/e2e-distinct-simulated-kserve-route

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.Service llm/e2e-distinct-simulated-kserve-workload-svc

llm

multus

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

AddedInterface

Add eth0 [10.134.0.36/23] from ovn-kubernetes

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.Deployment llm/e2e-distinct-simulated-kserve

llm

replicaset-controller

e2e-distinct-simulated-kserve-7b998b79d9

SuccessfulCreate

Created pod: e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

llm

deployment-controller

e2e-distinct-simulated-kserve

ScalingReplicaSet

Scaled up replica set e2e-distinct-simulated-kserve-7b998b79d9 from 0 to 1

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs

opendatahub

ai-gateway-controller-7cd9b8cccb-7bml5_0213b240-d5c7-44c6-ae44-81e5b0f26949

ai-gateway-controller-leader.opendatahub.io

LeaderElection

ai-gateway-controller-7cd9b8cccb-7bml5_0213b240-d5c7-44c6-ae44-81e5b0f26949 became leader

llm

kubelet

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

kubelet

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 611ms (611ms including waiting). Image size: 98346788 bytes.

llm

kubelet

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

Started

Container started

llm

kubelet

e2e-distinct-simulated-kserve-7b998b79d9-mgqz5

Created

Container created

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.PodMonitor llm/e2e-distinct-simulated-kserve-llmisvc-engine

llm

LLMInferenceServiceController

e2e-distinct-simulated

Created

Created v1.NetworkPolicy llm/e2e-distinct-simulated-prometheus-scraping

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs

llm

kubelet

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

multus

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

AddedInterface

Add eth0 [10.134.0.37/23] from ovn-kubernetes

llm

replicaset-controller

e2e-embedding-simulated-kserve-67bf594cd6

SuccessfulCreate

Created pod: e2e-embedding-simulated-kserve-67bf594cd6-r4s25

llm

deployment-controller

e2e-embedding-simulated-kserve

ScalingReplicaSet

Scaled up replica set e2e-embedding-simulated-kserve-67bf594cd6 from 0 to 1

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.Deployment llm/e2e-embedding-simulated-kserve

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.Service llm/e2e-embedding-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.HTTPRoute llm/e2e-embedding-simulated-kserve-route

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.DestinationRule llm/e2e-embedding-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.PodMonitor llm/e2e-embedding-simulated-kserve-llmisvc-engine

llm

kubelet

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 605ms (605ms including waiting). Image size: 98346788 bytes.

llm

kubelet

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

Created

Container created

llm

kubelet

e2e-embedding-simulated-kserve-67bf594cd6-r4s25

Started

Container started

llm

LLMInferenceServiceController

e2e-embedding-simulated

Created

Created v1.NetworkPolicy llm/e2e-embedding-simulated-prometheus-scraping

llm

replicaset-controller

e2e-trlp-test-simulated-kserve-69b588897d

SuccessfulCreate

Created pod: e2e-trlp-test-simulated-kserve-69b588897d-55lw7

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.DestinationRule llm/e2e-trlp-test-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.HTTPRoute llm/e2e-trlp-test-simulated-kserve-route

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.Service llm/e2e-trlp-test-simulated-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.Deployment llm/e2e-trlp-test-simulated-kserve

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs

llm

deployment-controller

e2e-trlp-test-simulated-kserve

ScalingReplicaSet

Scaled up replica set e2e-trlp-test-simulated-kserve-69b588897d from 0 to 1

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.PodMonitor llm/e2e-trlp-test-simulated-kserve-llmisvc-engine

llm

multus

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

AddedInterface

Add eth0 [10.134.0.38/23] from ovn-kubernetes

llm

kubelet

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Created

Created v1.NetworkPolicy llm/e2e-trlp-test-simulated-prometheus-scraping

llm

kubelet

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 776ms (776ms including waiting). Image size: 98346788 bytes.

llm

kubelet

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

Created

Container created

llm

kubelet

e2e-trlp-test-simulated-kserve-69b588897d-55lw7

Started

Container started

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.Deployment llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve

llm

kubelet

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.PodMonitor llm/e2e-unconab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.PodMonitor llm/eab60ef4d3a239b5143b412cab04acac3-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.DestinationRule llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.HTTPRoute llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-route

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.Service llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc

llm

deployment-controller

e2e-unconfigured-facebook-opt-125m-simulated-kserve

ScalingReplicaSet

Scaled up replica set e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc from 0 to 1

llm

replicaset-controller

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f4fc

SuccessfulCreate

Created pod: e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

llm

multus

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

AddedInterface

Add eth0 [10.134.0.39/23] from ovn-kubernetes

llm

kubelet

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

Started

Container started

llm

kubelet

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 642ms (642ms including waiting). Image size: 98346788 bytes.

llm

kubelet

e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mtl49

Created

Container created

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Created

Created v1.NetworkPolicy llm/e2e-unconfiab60ef4d3a239b5143b412cab04acac3-prometheus-scraping

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.Deployment llm/echo-simulator-kserve

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.Service llm/echo-simulator-kserve-workload-svc

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.Secret llm/echo-simulator-kserve-self-signed-certs

llm

deployment-controller

echo-simulator-kserve

ScalingReplicaSet

Scaled up replica set echo-simulator-kserve-64b6f8f96c from 0 to 1

llm

replicaset-controller

echo-simulator-kserve-64b6f8f96c

SuccessfulCreate

Created pod: echo-simulator-kserve-64b6f8f96c-tbfbx

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.HTTPRoute llm/echo-simulator-kserve-route

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.DestinationRule llm/echo-simulator-kserve-workload-svc

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine-default

llm

multus

echo-simulator-kserve-64b6f8f96c-tbfbx

AddedInterface

Add eth0 [10.132.0.56/23] from ovn-kubernetes

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.PodMonitor llm/echo-simulator-kserve-llmisvc-engine

llm

kubelet

echo-simulator-kserve-64b6f8f96c-tbfbx

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0"

llm

LLMInferenceServiceController

echo-simulator

Created

Created v1.NetworkPolicy llm/echo-simulator-prometheus-scraping

llm

kubelet

echo-simulator-kserve-64b6f8f96c-tbfbx

Started

Container started

llm

kubelet

echo-simulator-kserve-64b6f8f96c-tbfbx

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.9.0" in 2.397s (2.397s including waiting). Image size: 78208878 bytes.

llm

kubelet

echo-simulator-kserve-64b6f8f96c-tbfbx

Created

Container created

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.Deployment llm/facebook-opt-125m-simulated-kserve

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.DestinationRule llm/facebook-opt-125m-simulated-kserve-workload-svc

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

AddedInterface

Add eth0 [10.134.0.41/23] from ovn-kubernetes

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.HTTPRoute llm/facebook-opt-125m-simulated-kserve-route

llm

replicaset-controller

facebook-opt-125m-simulated-kserve-84b4ff656c

SuccessfulCreate

Created pod: facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.Service llm/facebook-opt-125m-simulated-kserve-workload-svc

llm

kubelet

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

multus

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

AddedInterface

Add eth0 [10.134.0.40/23] from ovn-kubernetes

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 3; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 1 to 3

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.PodMonitor llm/facebook-opt-125m-simulated-kserve-llmisvc-engine-default

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-rjj92

llm

deployment-controller

facebook-opt-125m-simulated-kserve

ScalingReplicaSet

Scaled up replica set facebook-opt-125m-simulated-kserve-84b4ff656c from 0 to 1

llm

kubelet

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 604ms (604ms including waiting). Image size: 98346788 bytes.

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Created

Created v1.NetworkPolicy llm/facebook-opt-125m-simulated-prometheus-scraping

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"

llm

kubelet

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

Started

Container started

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

AddedInterface

Add eth0 [10.133.0.31/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"

llm

kubelet

facebook-opt-125m-simulated-kserve-84b4ff656c-4t22t

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.904s (3.904s including waiting). Image size: 183555333 bytes.

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.145s (4.145s including waiting). Image size: 183555333 bytes.

llm

replicaset-controller

premium-simulated-simulated-premium-kserve-5f5c4b459f

SuccessfulCreate

Created pod: premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

llm

multus

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

AddedInterface

Add eth0 [10.134.0.42/23] from ovn-kubernetes

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.PodMonitor llm/p75d89895ad19b4ac62612d08bfe7f574-kserve-llmisvc-engine-default

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.DestinationRule llm/premium-simulated-simulated-premium-kserve-workload-svc

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.HTTPRoute llm/premium-simulated-simulated-premium-kserve-route

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.Service llm/premium-simulated-simulated-premium-kserve-workload-svc

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.Deployment llm/premium-simulated-simulated-premium-kserve

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs

llm

deployment-controller

premium-simulated-simulated-premium-kserve

ScalingReplicaSet

Scaled up replica set premium-simulated-simulated-premium-kserve-5f5c4b459f from 0 to 1

llm

kubelet

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 639ms (639ms including waiting). Image size: 98346788 bytes.

llm

kubelet

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

llm

kubelet

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

Started

Container started

llm

kubelet

premium-simulated-simulated-premium-kserve-5f5c4b459f-6j8cv

Created

Container created

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.PodMonitor llm/premium-simulated-simulated-premium-kserve-llmisvc-engine

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Created

Created v1.NetworkPolicy llm/premium-simulated-simulated-premium-prometheus-scraping
(x2)

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

Updated

Updated v1.Secret llm/e2e-distinct-2-simulated-kserve-self-signed-certs

llm

LLMInferenceServiceController

e2e-distinct-2-simulated

LLMInferenceServiceReady

LLMInferenceService [e2e-distinct-2-simulated] is Ready
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Unhealthy

Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

ProbeError

Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

ProbeError

Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Unhealthy

Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused

llm

LLMInferenceServiceController

e2e-distinct-simulated

Updated

Updated v1.Secret llm/e2e-distinct-simulated-kserve-self-signed-certs

llm

LLMInferenceServiceController

e2e-distinct-simulated

LLMInferenceServiceReady

LLMInferenceService [e2e-distinct-simulated] is Ready

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 3 to 4

llm

LLMInferenceServiceController

e2e-embedding-simulated

Updated

Updated v1.Secret llm/e2e-embedding-simulated-kserve-self-signed-certs

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Created

Container created

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

AddedInterface

Add eth0 [10.132.0.57/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

llm

LLMInferenceServiceController

e2e-embedding-simulated

LLMInferenceServiceReady

LLMInferenceService [e2e-embedding-simulated] is Ready

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

Updated

Updated v1.Secret llm/e2e-trlp-test-simulated-kserve-self-signed-certs

llm

LLMInferenceServiceController

e2e-trlp-test-simulated

LLMInferenceServiceReady

LLMInferenceService [e2e-trlp-test-simulated] is Ready
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

ProbeError

Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Unhealthy

Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

Updated

Updated v1.Secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs

llm

LLMInferenceServiceController

echo-simulator

Updated

Updated v1.Secret llm/echo-simulator-kserve-self-signed-certs

llm

LLMInferenceServiceController

e2e-unconfigured-facebook-opt-125m-simulated

LLMInferenceServiceReady

LLMInferenceService [e2e-unconfigured-facebook-opt-125m-simulated] is Ready

llm

LLMInferenceServiceController

echo-simulator

LLMInferenceServiceReady

LLMInferenceService [echo-simulator] is Ready

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

Updated

Updated v1.Secret llm/facebook-opt-125m-simulated-kserve-self-signed-certs

llm

LLMInferenceServiceController

facebook-opt-125m-simulated

LLMInferenceServiceReady

LLMInferenceService [facebook-opt-125m-simulated] is Ready

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 4 to 6

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 6; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-st9jq

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-smv2q

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

AddedInterface

Add eth0 [10.134.0.43/23] from ovn-kubernetes

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

AddedInterface

Add eth0 [10.133.0.32/23] from ovn-kubernetes

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

Updated

Updated v1.Secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Created

Container created

llm

LLMInferenceServiceController

premium-simulated-simulated-premium

LLMInferenceServiceReady

LLMInferenceService [premium-simulated-simulated-premium] is Ready

kuadrant-system

replicaset-controller

authorino-5cb7d797b8

SuccessfulCreate

Created pod: authorino-5cb7d797b8-rjd4x

kuadrant-system

kubelet

authorino-5cb7d797b8-rjd4x

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0"

kuadrant-system

multus

authorino-5cb7d797b8-rjd4x

AddedInterface

Add eth0 [10.132.0.58/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-5cb7d797b8-rjd4x

Created

Container created

kuadrant-system

kubelet

authorino-6f7c4f5d87-28j47

Killing

Stopping container authorino

kuadrant-system

kubelet

authorino-5cb7d797b8-rjd4x

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0" in 1.846s (1.846s including waiting). Image size: 204096810 bytes.

kuadrant-system

kubelet

authorino-5cb7d797b8-rjd4x

Started

Container started
(x2)

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

(combined from similar events): Scaled down replica set authorino-6f7c4f5d87 from 1 to 0

kuadrant-system

replicaset-controller

authorino-6f7c4f5d87

SuccessfulDelete

Deleted pod: authorino-6f7c4f5d87-28j47
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

ProbeError

Startup probe error: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

Unhealthy

Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-st9jq

ProbeError

Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Unhealthy

Startup probe failed: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused

openshift-ingress

deployment-controller

e2e-worker-w2-fba151-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w2-fba151-openshift-default-d5688d5cc from 0 to 1

openshift-ingress

replicaset-controller

e2e-worker-w2-fba151-openshift-default-d5688d5cc

SuccessfulCreate

Created pod: e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

openshift-ingress

gateway_labeler_controller

e2e-worker-w2-fba151

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-fba151

openshift-ingress

gateway_labeler_controller

e2e-worker-w3-82c724

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-82c724

openshift-ingress

replicaset-controller

e2e-worker-w3-82c724-openshift-default-67ff8c8875

SuccessfulCreate

Created pod: e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

openshift-ingress

deployment-controller

e2e-worker-w3-82c724-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w3-82c724-openshift-default-67ff8c8875 from 0 to 1

openshift-ingress

multus

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

AddedInterface

Add eth0 [10.132.0.60/23] from ovn-kubernetes

openshift-ingress

multus

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

AddedInterface

Add eth0 [10.132.0.61/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-processing-e2e-worker-w2-fba151

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-worker-w2-fba151-59d7cc774 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-worker-w2-fba151

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67 from 0 to 1

openshift-ingress

deployment-controller

e2e-worker-w1-ee1947-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67

SuccessfulCreate

Created pod: payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

openshift-ingress

gateway_labeler_controller

e2e-worker-w1-ee1947

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-ee1947
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w2-fba151-59d7cc774

FailedCreate

Error creating: pods "payload-processing-e2e-worker-w2-fba151-59d7cc774-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-fba151: serviceaccount "payload-processing-e2e-worker-w2-fba151" not found
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67

FailedCreate

Error creating: pods "payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-fba151: serviceaccount "payload-processing-e2e-worker-w2-fba151" not found

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Started

Container started

openshift-ingress

multus

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

AddedInterface

Add eth0 [10.134.0.45/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

AddedInterface

Add eth0 [10.132.0.62/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Created

Container created

openshift-ingress

multus

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

AddedInterface

Add eth0 [10.134.0.44/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w2-fba151-59d7cc774

SuccessfulCreate

Created pod: payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

openshift-ingress

deployment-controller

e2e-worker-w4-50429e-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w4-50429e-openshift-default-755c7b9cf9 from 0 to 1

openshift-ingress

gateway_labeler_controller

e2e-worker-w4-50429e

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-50429e

openshift-ingress

replicaset-controller

e2e-worker-w4-50429e-openshift-default-755c7b9cf9

SuccessfulCreate

Created pod: e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

openshift-ingress

multus

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

AddedInterface

Add eth0 [10.134.0.46/23] from ovn-kubernetes

openshift-ingress

deployment-controller

e2e-worker-w0-092f8c-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w0-092f8c-openshift-default-fffbc45cc from 0 to 1

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

openshift-ingress

gateway_labeler_controller

e2e-worker-w0-092f8c

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-092f8c

openshift-ingress

deployment-controller

payload-processing-e2e-worker-w3-82c724

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8 from 0 to 1
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff

FailedCreate

Error creating: pods "payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-82c724: serviceaccount "payload-processing-e2e-worker-w3-82c724" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff

SuccessfulCreate

Created pod: payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

openshift-ingress

deployment-controller

payload-pre-processing-e2e-worker-w3-82c724

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff from 0 to 1

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

AddedInterface

Add eth0 [10.132.0.63/23] from ovn-kubernetes
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8

FailedCreate

Error creating: pods "payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-82c724: serviceaccount "payload-processing-e2e-worker-w3-82c724" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8

SuccessfulCreate

Created pod: payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Pulling

Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"

openshift-ingress

kubelet

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Pulling

Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"

openshift-ingress

multus

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

AddedInterface

Add eth0 [10.132.0.65/23] from ovn-kubernetes

openshift-ingress

multus

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

AddedInterface

Add eth0 [10.132.0.64/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Pulled

Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.267s (3.267s including waiting). Image size: 125353963 bytes.

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Pulled

Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.18s (3.18s including waiting). Image size: 125353963 bytes.
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6

FailedCreate

Error creating: pods "payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-ee1947: serviceaccount "payload-processing-e2e-worker-w1-ee1947" not found
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc

FailedCreate

Error creating: pods "payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-ee1947: serviceaccount "payload-processing-e2e-worker-w1-ee1947" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc

SuccessfulCreate

Created pod: payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

openshift-ingress

deployment-controller

payload-processing-e2e-worker-w1-ee1947

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-worker-w1-ee1947-74f4dc77d6 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-worker-w1-ee1947

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc from 0 to 1

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

AddedInterface

Add eth0 [10.132.0.66/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6

SuccessfulCreate

Created pod: payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Created

Container created

openshift-ingress

multus

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

AddedInterface

Add eth0 [10.134.0.49/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

deployment-controller

payload-processing-e2e-worker-w4-50429e

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-worker-w4-50429e-7bfb595764 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-worker-w4-50429e

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59

SuccessfulCreate

Created pod: payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59

FailedCreate

Error creating: pods "payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-50429e: serviceaccount "payload-processing-e2e-worker-w4-50429e" not found
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w4-50429e-7bfb595764

FailedCreate

Error creating: pods "payload-processing-e2e-worker-w4-50429e-7bfb595764-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-50429e: serviceaccount "payload-processing-e2e-worker-w4-50429e" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w4-50429e-7bfb595764

SuccessfulCreate

Created pod: payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

AddedInterface

Add eth0 [10.132.0.67/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

AddedInterface

Add eth0 [10.134.0.50/23] from ovn-kubernetes
(x2)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Created

Container created

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

ProbeError

Readiness probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused body:
(x2)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

ProbeError

Readiness probe error: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused body:
(x2)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Created

Container created

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Unhealthy

Readiness probe failed: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Unhealthy

Readiness probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc

FailedCreate

Error creating: pods "payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-092f8c: serviceaccount "payload-processing-e2e-worker-w0-092f8c" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc

SuccessfulCreate

Created pod: payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w0-092f8c-7f979b7686

FailedCreate

Error creating: pods "payload-processing-e2e-worker-w0-092f8c-7f979b7686-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-092f8c: serviceaccount "payload-processing-e2e-worker-w0-092f8c" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-worker-w0-092f8c-7f979b7686

SuccessfulCreate

Created pod: payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

openshift-ingress

deployment-controller

payload-processing-e2e-worker-w0-092f8c

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-worker-w0-092f8c-7f979b7686 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-worker-w0-092f8c

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc from 0 to 1
(x5)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

ProbeError

Startup probe error: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Created

Container created

openshift-ingress

multus

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

AddedInterface

Add eth0 [10.132.0.69/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
(x5)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Unhealthy

Startup probe failed: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused

openshift-ingress

multus

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

AddedInterface

Add eth0 [10.132.0.68/23] from ovn-kubernetes
(x5)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Unhealthy

Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

ProbeError

Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Started

Container started
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w4-50429e-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w4-50429e-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Unhealthy

Startup probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

ProbeError

Startup probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused body:
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Started

Container started
(x5)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Unhealthy

Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x5)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

ProbeError

Startup probe error: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused body:
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w4-50429e-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w4-50429e-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Started

Container started

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x5)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w2-fba151-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

replicaset-controller

e2e-worker-w3-82c724-openshift-default-67ff8c8875

SuccessfulCreate

Created pod: e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj
(x5)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w2-fba151-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Started

Container started
(x5)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Unhealthy

Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

ProbeError

Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused body:

openshift-ingress

multus

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

AddedInterface

Add eth0 [10.133.0.35/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Created

Container created
(x6)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

ProbeError

Startup probe error: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Unhealthy

Startup probe failed: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

AddedInterface

Add eth0 [10.134.0.58/23] from ovn-kubernetes

openshift-ingress

deployment-controller

e2e-worker-w1-ee1947-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f from 1 to 2

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

SuccessfulRescale

New size: 2; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Started

Container started
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

ProbeError

Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Unhealthy

Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

openshift-ingress

deployment-controller

e2e-worker-w0-092f8c-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w0-092f8c-openshift-default-fffbc45cc from 1 to 2

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

SuccessfulRescale

New size: 2; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

AddedInterface

Add eth0 [10.133.0.40/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Started

Container started

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Created

Container created

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled down replica set maas-default-gateway-openshift-default-8b64f9ff9 from 6 to 5

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulDelete

Deleted pod: maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 5; reason: All metrics below target

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-bvxs5

Killing

Stopping container istio-proxy
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

ProbeError

Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Unhealthy

Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

openshift-ingress

deployment-controller

e2e-worker-w1-ee1947-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f from 2 to 3

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

SuccessfulRescale

New size: 3; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Started

Container started

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

AddedInterface

Add eth0 [10.133.0.41/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Created

Container created
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Unhealthy

Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

ProbeError

Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused body:

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulDelete

Deleted pod: maas-default-gateway-openshift-default-8b64f9ff9-rjj92

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-rjj92

Killing

Stopping container istio-proxy

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled down replica set maas-default-gateway-openshift-default-8b64f9ff9 from 5 to 4

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 4; reason: All metrics below target

opendatahub

multus

test-curl-7370-7bea52

AddedInterface

Add eth0 [10.132.0.71/23] from ovn-kubernetes

opendatahub

kubelet

test-curl-7370-7bea52

Pulling

Pulling image "registry.access.redhat.com/ubi9/ubi-minimal:9.5"

opendatahub

kubelet

test-curl-7370-7bea52

Created

Container created

opendatahub

kubelet

test-curl-7370-7bea52

Started

Container started

opendatahub

kubelet

test-curl-7370-7bea52

Pulled

Successfully pulled image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" in 3.618s (3.618s including waiting). Image size: 106100986 bytes.

opendatahub

kubelet

test-curl-7370-7bea52

Killing

Stopping container test-curl-7370-7bea52

opendatahub

multus

test-curl-7370-aff15b

AddedInterface

Add eth0 [10.132.0.72/23] from ovn-kubernetes

opendatahub

kubelet

test-curl-7370-aff15b

Created

Container created

opendatahub

kubelet

test-curl-7370-aff15b

Started

Container started

opendatahub

kubelet

test-curl-7370-aff15b

Pulled

Container image "registry.access.redhat.com/ubi9/ubi-minimal:9.5" already present on machine and can be accessed by the pod

opendatahub

kubelet

test-curl-7370-aff15b

Killing

Stopping container test-curl-7370-aff15b

openshift-ingress

gateway_labeler_controller

e2e-extmod-a-5d4032

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-a-5d4032

openshift-ingress

replicaset-controller

e2e-extmod-a-5d4032-openshift-default-5c878f7d45

SuccessfulCreate

Created pod: e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

openshift-ingress

deployment-controller

e2e-extmod-a-5d4032-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-a-5d4032-openshift-default-5c878f7d45 from 0 to 1

openshift-ingress

multus

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

AddedInterface

Add eth0 [10.132.0.73/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745

SuccessfulCreate

Created pod: e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

openshift-ingress

gateway_labeler_controller

e2e-extmod-a-8ecbc5

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-a-8ecbc5

openshift-ingress

deployment-controller

e2e-extmod-a-8ecbc5-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-a-8ecbc5-openshift-default-57b54b6745 from 0 to 1

openshift-ingress

multus

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

AddedInterface

Add eth0 [10.132.0.74/23] from ovn-kubernetes

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-5d4032-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-a-8ecbc5

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-a-8ecbc5-776655854d from 0 to 1

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-5d4032-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-8ecbc5-776655854d

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-8ecbc5-776655854d

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-a-8ecbc5-776655854d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-8ecbc5: serviceaccount "payload-processing-e2e-extmod-a-8ecbc5" not found

openshift-ingress

multus

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

AddedInterface

Add eth0 [10.132.0.75/23] from ovn-kubernetes
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-8ecbc5: serviceaccount "payload-processing-e2e-extmod-a-8ecbc5" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-a-8ecbc5

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9 from 0 to 1

openshift-ingress

multus

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

AddedInterface

Add eth0 [10.132.0.76/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-a-5d4032

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-a-5d4032-6d7c8b496d from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-a-5d4032

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759 from 0 to 1

openshift-ingress

multus

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

AddedInterface

Add eth0 [10.132.0.77/23] from ovn-kubernetes
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-5d4032: serviceaccount "payload-processing-e2e-extmod-a-5d4032" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-5d4032: serviceaccount "payload-processing-e2e-extmod-a-5d4032" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

Started

Container started

openshift-ingress

multus

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

AddedInterface

Add eth0 [10.132.0.78/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

SuccessfulRescale

New size: 5; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

openshift-ingress

deployment-controller

e2e-worker-w1-ee1947-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f from 3 to 5

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

AddedInterface

Add eth0 [10.134.0.61/23] from ovn-kubernetes

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

AddedInterface

Add eth0 [10.132.0.79/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Started

Container started

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Started

Container started

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled down replica set maas-default-gateway-openshift-default-8b64f9ff9 from 4 to 3

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 3; reason: All metrics below target

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-2tbrd

Killing

Stopping container istio-proxy
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

ProbeError

Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Unhealthy

Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulDelete

Deleted pod: maas-default-gateway-openshift-default-8b64f9ff9-2tbrd
(x7)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

ProbeError

Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Unhealthy

Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Created

Container created

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled down replica set maas-default-gateway-openshift-default-8b64f9ff9 from 3 to 2

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 2; reason: All metrics below target

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

e2e-worker-w2-fba151-openshift-default-d5688d5cc

SuccessfulCreate

Created pod: e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

openshift-ingress

multus

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

AddedInterface

Add eth0 [10.133.0.44/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulDelete

Deleted pod: maas-default-gateway-openshift-default-8b64f9ff9-smv2q

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-smv2q

Killing

Stopping container istio-proxy
(x7)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

ProbeError

Startup probe error: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Unhealthy

Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x5)

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Unhealthy

Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused

openshift-ingress

gateway_labeler_controller

e2e-extmod-b-1bf4bf

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-b-1bf4bf
(x5)

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

ProbeError

Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused body:

openshift-ingress

replicaset-controller

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485

SuccessfulCreate

Created pod: e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

openshift-ingress

deployment-controller

e2e-extmod-b-1bf4bf-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-b-1bf4bf-openshift-default-7fc4875485 from 0 to 1

openshift-ingress

multus

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

AddedInterface

Add eth0 [10.132.0.82/23] from ovn-kubernetes
(x2)

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

e2e-extmod-a-6d170d-openshift-default-86667cff57

SuccessfulCreate

Created pod: e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

openshift-ingress

gateway_labeler_controller

e2e-extmod-a-6d170d

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-a-6d170d

openshift-ingress

deployment-controller

e2e-extmod-a-6d170d-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-a-6d170d-openshift-default-86667cff57 from 0 to 1

openshift-ingress

multus

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

AddedInterface

Add eth0 [10.132.0.83/23] from ovn-kubernetes
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-8ecbc5-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-8ecbc5-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Unhealthy

Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

ProbeError

Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused body:
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-1bf4bf: serviceaccount "payload-processing-e2e-extmod-b-1bf4bf" not found
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-1bf4bf: serviceaccount "payload-processing-e2e-extmod-b-1bf4bf" not found

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-b-1bf4bf

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-b-1bf4bf

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5 from 0 to 1

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

AddedInterface

Add eth0 [10.134.0.63/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

AddedInterface

Add eth0 [10.132.0.84/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

Started

Container started

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

Created

Container created

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

openshift-ingress

gateway_labeler_controller

e2e-extmod-b-59c311

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-b-59c311

openshift-ingress

replicaset-controller

e2e-extmod-b-59c311-openshift-default-5dbbb75c65

SuccessfulCreate

Created pod: e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

openshift-ingress

multus

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

AddedInterface

Add eth0 [10.132.0.85/23] from ovn-kubernetes

openshift-ingress

deployment-controller

e2e-extmod-b-59c311-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-b-59c311-openshift-default-5dbbb75c65 from 0 to 1
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-5d4032-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-5d4032-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-6d170d: serviceaccount "payload-processing-e2e-extmod-a-6d170d" not found

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-a-6d170d

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8 from 0 to 1
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-a-6d170d: serviceaccount "payload-processing-e2e-extmod-a-6d170d" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-a-6d170d

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c from 0 to 1

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Pulling

Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable"

openshift-ingress

multus

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

AddedInterface

Add eth0 [10.132.0.86/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

Started

Container started

openshift-ingress

multus

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

AddedInterface

Add eth0 [10.133.0.45/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Pulled

Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.825s (2.825s including waiting). Image size: 125353963 bytes.

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Started

Container started

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-8ecbc5-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-8ecbc5-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-b-59c311

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-b-59c311-6987c8b78 from 0 to 1
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-59c311: serviceaccount "payload-processing-e2e-extmod-b-59c311" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-b-59c311

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9 from 0 to 1
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-59c311-6987c8b78

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-b-59c311-6987c8b78-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-59c311: serviceaccount "payload-processing-e2e-extmod-b-59c311" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-59c311-6987c8b78

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

AddedInterface

Add eth0 [10.132.0.87/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

Started

Container started

openshift-ingress

multus

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

AddedInterface

Add eth0 [10.134.0.64/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Created

Container created

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

AddedInterface

Add eth0 [10.132.0.89/23] from ovn-kubernetes

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

AddedInterface

Add eth0 [10.132.0.88/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

AddedInterface

Add eth0 [10.134.0.65/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

openshift-ingress

multus

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

AddedInterface

Add eth0 [10.133.0.47/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w1-ee1947-openshift-default

SuccessfulRescale

New size: 10; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

deployment-controller

e2e-worker-w1-ee1947-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f from 5 to 10

openshift-ingress

replicaset-controller

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f

SuccessfulCreate

(combined from similar events): Created pod: e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Started

Container started

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Started

Container started

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Started

Container started

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Created

Container created

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Started

Container started

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Started

Container started
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

ProbeError

Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Unhealthy

Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Unhealthy

Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Unhealthy

Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

ProbeError

Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

ProbeError

Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

ProbeError

Startup probe error: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Unhealthy

Startup probe failed: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Unhealthy

Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

ProbeError

Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused body:

openshift-ingress

deployment-controller

e2e-worker-w3-82c724-openshift-default

ScalingReplicaSet

Scaled down replica set e2e-worker-w3-82c724-openshift-default-67ff8c8875 from 2 to 1

openshift-ingress

replicaset-controller

e2e-worker-w3-82c724-openshift-default-67ff8c8875

SuccessfulDelete

Deleted pod: e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

SuccessfulRescale

New size: 1; reason: All metrics below target
(x8)

models-as-a-service

replicaset-controller

payload-processing-external-model-5bcd5df458

FailedCreate

Error creating: pods "payload-processing-external-model-5bcd5df458-" is forbidden: error looking up service account models-as-a-service/payload-processing-external-model: serviceaccount "payload-processing-external-model" not found

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled up replica set payload-processing-external-model-5bcd5df458 from 0 to 1

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Killing

Stopping container istio-proxy

models-as-a-service

replicaset-controller

payload-processing-external-model-5bcd5df458

SuccessfulCreate

Created pod: payload-processing-external-model-5bcd5df458-7z9tf

models-as-a-service

kubelet

payload-processing-external-model-5bcd5df458-7z9tf

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

models-as-a-service

kubelet

payload-processing-external-model-5bcd5df458-7z9tf

Created

Container created

models-as-a-service

multus

payload-processing-external-model-5bcd5df458-7z9tf

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

models-as-a-service

kubelet

payload-processing-external-model-5bcd5df458-7z9tf

Started

Container started

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-v42bj

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

models-as-a-service

replicaset-controller

payload-processing-external-model-7f5c68695b

SuccessfulCreate

Created pod: payload-processing-external-model-7f5c68695b-7z22d

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled up replica set payload-processing-external-model-7f5c68695b from 0 to 1

models-as-a-service

multus

payload-processing-external-model-7f5c68695b-7z22d

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

models-as-a-service

kubelet

payload-processing-external-model-7f5c68695b-7z22d

Created

Container created

models-as-a-service

kubelet

payload-processing-external-model-7f5c68695b-7z22d

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

models-as-a-service

kubelet

payload-processing-external-model-7f5c68695b-7z22d

Started

Container started

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

AddedInterface

Add eth0 [10.134.0.67/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Created

Container created

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Started

Container started

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

AddedInterface

Add eth0 [10.132.0.91/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Created

Container created

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Started

Container started

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Unhealthy

Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

ProbeError

Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused body:
(x8)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

ProbeError

Startup probe error: Get "http://10.132.0.91:15021/healthz/ready": dial tcp 10.132.0.91:15021: connect: connection refused body:

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Unhealthy

Readiness probe failed: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

ProbeError

Readiness probe error: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:
(x8)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Unhealthy

Startup probe failed: Get "http://10.132.0.91:15021/healthz/ready": dial tcp 10.132.0.91:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Created

Container created
(x2)

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Started

Container started

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-6d170d-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

models-as-a-service

replicaset-controller

payload-processing-external-model-5bcd5df458

SuccessfulDelete

Deleted pod: payload-processing-external-model-5bcd5df458-7z9tf

models-as-a-service

kubelet

payload-processing-external-model-5bcd5df458-7z9tf

Killing

Stopping container payload-processing

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled down replica set payload-processing-external-model-5bcd5df458 from 1 to 0

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-6d170d-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x5)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Unhealthy

Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

ProbeError

Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Unhealthy

Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

ProbeError

Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused body:

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

Started

Container started

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 2 to 4

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

AddedInterface

Add eth0 [10.133.0.51/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

Created

Container created

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

AddedInterface

Add eth0 [10.133.0.52/23] from ovn-kubernetes
(x2)

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 4; reason: cpu resource utilization (percentage of request) above target
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-1bf4bf-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-1bf4bf-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

ProbeError

Startup probe error: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused body:
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

Unhealthy

Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-j6k8d

ProbeError

Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused body:
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fd2nl

Unhealthy

Startup probe failed: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Unhealthy

Readiness probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

ProbeError

Readiness probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused body:

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 8; reason:
(x2)

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Created

Container created

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-8b64f9ff9-hgwbk
(x2)

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Started

Container started

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

AddedInterface

Add eth0 [10.134.0.69/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

AddedInterface

Add eth0 [10.132.0.93/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

AddedInterface

Add eth0 [10.132.0.92/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

Created

Container created

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

AddedInterface

Add eth0 [10.133.0.53/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

Started

Container started

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x5)

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Unhealthy

Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

ProbeError

Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused body:
(x4)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-6d170d-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-a-6d170d-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

Unhealthy

Startup probe failed: Get "http://10.132.0.93:15021/healthz/ready": dial tcp 10.132.0.93:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

Unhealthy

Startup probe failed: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

Unhealthy

Startup probe failed: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

ProbeError

Startup probe error: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-hgwbk

ProbeError

Startup probe error: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-fms7t

Unhealthy

Startup probe failed: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-jx8n4

ProbeError

Startup probe error: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-26xvk

ProbeError

Startup probe error: Get "http://10.132.0.93:15021/healthz/ready": dial tcp 10.132.0.93:15021: connect: connection refused body:
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-59c311-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-59c311-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

AddedInterface

Add eth0 [10.134.0.70/23] from ovn-kubernetes

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

New size: 9; reason:

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

Created

Container created
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-1bf4bf-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-1bf4bf-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

Started

Container started

openshift-ingress

gateway_labeler_controller

e2e-extmod-b-951893

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-extmod-b-951893

openshift-ingress

multus

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

AddedInterface

Add eth0 [10.133.0.56/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

e2e-extmod-b-951893-openshift-default-586cb5d9bf

SuccessfulCreate

Created pod: e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

openshift-ingress

deployment-controller

e2e-extmod-b-951893-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-extmod-b-951893-openshift-default-586cb5d9bf from 0 to 1
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

Unhealthy

Startup probe failed: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused
(x7)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-sr9mv

ProbeError

Startup probe error: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused body:

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled up replica set payload-processing-external-model-8b89f9dc6 from 0 to 1

models-as-a-service

replicaset-controller

payload-processing-external-model-8b89f9dc6

SuccessfulCreate

Created pod: payload-processing-external-model-8b89f9dc6-zbkqz

models-as-a-service

multus

payload-processing-external-model-8b89f9dc6-zbkqz

AddedInterface

Add eth0 [10.133.0.58/23] from ovn-kubernetes

models-as-a-service

kubelet

payload-processing-external-model-8b89f9dc6-zbkqz

Started

Container started

models-as-a-service

kubelet

payload-processing-external-model-8b89f9dc6-zbkqz

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

models-as-a-service

kubelet

payload-processing-external-model-8b89f9dc6-zbkqz

Created

Container created
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-59c311-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-59c311-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7

FailedCreate

Error creating: pods "payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-951893: serviceaccount "payload-processing-e2e-extmod-b-951893" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-extmod-b-951893

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7 from 0 to 1

openshift-ingress

deployment-controller

payload-processing-e2e-extmod-b-951893

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-extmod-b-951893-5d6556fd68 from 0 to 1
(x7)

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-951893-5d6556fd68

FailedCreate

Error creating: pods "payload-processing-e2e-extmod-b-951893-5d6556fd68-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-extmod-b-951893: serviceaccount "payload-processing-e2e-extmod-b-951893" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7

SuccessfulCreate

Created pod: payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

openshift-ingress

replicaset-controller

payload-processing-e2e-extmod-b-951893-5d6556fd68

SuccessfulCreate

Created pod: payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

openshift-ingress

multus

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

AddedInterface

Add eth0 [10.133.0.59/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

multus

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

AddedInterface

Add eth0 [10.134.0.71/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Created

Container created

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

Created

Container created

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Started

Container started

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Killing

Stopping container payload-processing

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-951893-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-951893-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w1-ee1947-bdbb8c7bc-jvjjq

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x2)

openshift-ingress

kubelet

payload-processing-e2e-worker-w1-ee1947-74f4dc77d6-z7f9w

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

horizontal-pod-autoscaler

maas-default-gateway-openshift-default

SuccessfulRescale

(combined from similar events): New size: 10; reason: cpu resource utilization (percentage of request) above target
(x5)

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-8b64f9ff9

SuccessfulCreate

(combined from similar events): Created pod: maas-default-gateway-openshift-default-8b64f9ff9-dh692
(x3)

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

(combined from similar events): Scaled up replica set maas-default-gateway-openshift-default-8b64f9ff9 from 9 to 10

openshift-ingress

multus

maas-default-gateway-openshift-default-8b64f9ff9-dh692

AddedInterface

Add eth0 [10.134.0.73/23] from ovn-kubernetes

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-dh692

Created

Container created

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-dh692

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-dh692

Started

Container started
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-dh692

ProbeError

Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused body:
(x8)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-8b64f9ff9-dh692

Unhealthy

Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-2t9nc

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-dvcgm

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rwndz

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-zlp2s

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-5jkwd

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-rjvs8

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-llmgw

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-6rqkd

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-9mzkh

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w1-ee1947-openshift-default-7f6dcb4f6f-549gb

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:
(x2)

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Started

Container started
(x2)

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Created

Container created
(x6)

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

ProbeError

Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Unhealthy

Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-951893-openshift-default

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

e2e-extmod-b-951893-openshift-default

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-extmod-a-8ecbc5-openshift-default-57b54b6745-lh58v

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:
(x2)

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-8ecbc5-776655854d-p7qp2

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-8ecbc5-7df459b5c9-f2hdp

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-1bf4bf-5bcc9466c5-b67d2

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-1bf4bf-6699d6c4bd-5rfjn

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-5d4032-6d7c8b496d-5rzsv

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-5d4032-854b9cd759-rcv54

Killing

Stopping container payload-pre-processing
(x9)

models-as-a-service

kubelet

payload-processing-external-model-7f5c68695b-7z22d

FailedMount

MountVolume.SetUp failed for volume "provider-credentials" : secret "agc-cred-243e19a17a73" not found

openshift-ingress

deployment-controller

e2e-worker-w2-fba151-openshift-default

ScalingReplicaSet

Scaled down replica set e2e-worker-w2-fba151-openshift-default-d5688d5cc from 2 to 1

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

Killing

Stopping container istio-proxy

openshift-ingress

replicaset-controller

e2e-worker-w2-fba151-openshift-default-d5688d5cc

SuccessfulDelete

Deleted pod: e2e-worker-w2-fba151-openshift-default-d5688d5cc-vkhpd

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w2-fba151-openshift-default

SuccessfulRescale

New size: 1; reason: All metrics below target

openshift-ingress

kubelet

e2e-extmod-a-5d4032-openshift-default-5c878f7d45-67qnk

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-extmod-b-1bf4bf-openshift-default-7fc4875485-kxt4j

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-59c311-6987c8b78-vpbrv

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-59c311-77bc87bfc9-xvvgl

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-worker-w4-50429e-7bfb595764-n2l4c

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-extmod-b-59c311-openshift-default-5dbbb75c65-bf52c

Killing

Stopping container istio-proxy

models-as-a-service

replicaset-controller

payload-processing-external-model-8b89f9dc6

SuccessfulDelete

Deleted pod: payload-processing-external-model-8b89f9dc6-zbkqz

models-as-a-service

kubelet

payload-processing-external-model-8b89f9dc6-zbkqz

Killing

Stopping container payload-processing
(x22)

models-as-a-service

kubelet

payload-processing-external-model-8b89f9dc6-zbkqz

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled up replica set payload-processing-external-model-df4cf9975 from 0 to 1

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled down replica set payload-processing-external-model-8b89f9dc6 from 1 to 0

models-as-a-service

replicaset-controller

payload-processing-external-model-df4cf9975

SuccessfulCreate

Created pod: payload-processing-external-model-df4cf9975-hk75w

models-as-a-service

kubelet

payload-processing-external-model-df4cf9975-hk75w

Created

Container created

models-as-a-service

kubelet

payload-processing-external-model-df4cf9975-hk75w

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod

models-as-a-service

multus

payload-processing-external-model-df4cf9975-hk75w

AddedInterface

Add eth0 [10.132.0.97/23] from ovn-kubernetes

models-as-a-service

kubelet

payload-processing-external-model-df4cf9975-hk75w

Started

Container started
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w4-50429e-77f4c85c59-wphrx

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

kubelet

payload-processing-e2e-worker-w3-82c724-6d4b6d5bb8-8cdsz

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Killing

Stopping container payload-pre-processing

models-as-a-service

replicaset-controller

payload-processing-external-model-7f5c68695b

SuccessfulDelete

Deleted pod: payload-processing-external-model-7f5c68695b-7z22d

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled down replica set payload-processing-external-model-7f5c68695b from 1 to 0

models-as-a-service

kubelet

payload-processing-external-model-7f5c68695b-7z22d

Killing

Stopping container payload-processing

openshift-ingress

replicaset-controller

e2e-worker-w4-50429e-openshift-default-755c7b9cf9

SuccessfulCreate

Created pod: e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

openshift-ingress

deployment-controller

e2e-worker-w4-50429e-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w4-50429e-openshift-default-755c7b9cf9 from 1 to 2

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w4-50429e-openshift-default

SuccessfulRescale

New size: 2; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

multus

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

AddedInterface

Add eth0 [10.132.0.99/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Created

Container created

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Started

Container started
(x6)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

ProbeError

Startup probe error: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Unhealthy

Startup probe failed: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w3-82c724-76bcd64bff-94mnv

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-xsv87

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Killing

Stopping container istio-proxy

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulDelete

Deleted pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

SuccessfulRescale

New size: 3; reason: All metrics below target

openshift-ingress

deployment-controller

e2e-worker-w0-092f8c-openshift-default

ScalingReplicaSet

Scaled down replica set e2e-worker-w0-092f8c-openshift-default-fffbc45cc from 4 to 3

openshift-ingress

kubelet

e2e-worker-w4-50429e-openshift-default-755c7b9cf9-h4tsh

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-n7r6b

Killing

Stopping container istio-proxy

openshift-ingress

replicaset-controller

e2e-worker-w3-82c724-openshift-default-67ff8c8875

SuccessfulCreate

Created pod: e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n
(x2)

openshift-ingress

deployment-controller

e2e-worker-w3-82c724-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w3-82c724-openshift-default-67ff8c8875 from 1 to 2
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w3-82c724-openshift-default

SuccessfulRescale

New size: 2; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Started

Container started

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Created

Container created

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

multus

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

AddedInterface

Add eth0 [10.134.0.76/23] from ovn-kubernetes

models-as-a-service

deployment-controller

payload-processing-external-model

ScalingReplicaSet

Scaled up replica set payload-processing-external-model-7489856dc7 from 0 to 1

models-as-a-service

replicaset-controller

payload-processing-external-model-7489856dc7

SuccessfulCreate

Created pod: payload-processing-external-model-7489856dc7-h45cw

models-as-a-service

kubelet

payload-processing-external-model-7489856dc7-h45cw

Created

Container created

models-as-a-service

kubelet

payload-processing-external-model-7489856dc7-h45cw

Started

Container started

models-as-a-service

multus

payload-processing-external-model-7489856dc7-h45cw

AddedInterface

Add eth0 [10.132.0.101/23] from ovn-kubernetes

models-as-a-service

kubelet

payload-processing-external-model-7489856dc7-h45cw

Pulled

Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod
(x4)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Unhealthy

Startup probe failed: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

ProbeError

Startup probe error: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused body:

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-k5vjf

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w3-82c724-openshift-default-67ff8c8875-5vs5n

Killing

Stopping container istio-proxy

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

SuccessfulRescale

New size: 2; reason: All metrics below target

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Killing

Stopping container istio-proxy

openshift-ingress

deployment-controller

e2e-worker-w0-092f8c-openshift-default

ScalingReplicaSet

Scaled down replica set e2e-worker-w0-092f8c-openshift-default-fffbc45cc from 3 to 2

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulDelete

Deleted pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

ProbeError

Readiness probe error: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-z97xr

Unhealthy

Readiness probe failed: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

openshift-ingress

kubelet

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Killing

Stopping container payload-pre-processing

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

openshift-ingress

replicaset-controller

e2e-worker-w0-092f8c-openshift-default-fffbc45cc

SuccessfulCreate

Created pod: e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod
(x2)

openshift-ingress

deployment-controller

e2e-worker-w0-092f8c-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w0-092f8c-openshift-default-fffbc45cc from 2 to 4
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w0-092f8c-openshift-default

SuccessfulRescale

New size: 4; reason: cpu resource utilization (percentage of request) above target

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

AddedInterface

Add eth0 [10.134.0.77/23] from ovn-kubernetes

openshift-ingress

multus

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

AddedInterface

Add eth0 [10.132.0.104/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Started

Container started

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Created

Container created

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Created

Container created

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Started

Container started
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w0-092f8c-679c8b7dbc-mcm7c

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x2)

openshift-ingress

kubelet

payload-processing-e2e-worker-w0-092f8c-7f979b7686-zq4hl

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

ProbeError

Startup probe error: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused body:
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Unhealthy

Startup probe failed: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Unhealthy

Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

ProbeError

Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused body:

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-pjdng

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-f5sjs

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-77c47

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w0-092f8c-openshift-default-fffbc45cc-x4d9x

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x25)

models-as-a-service

kubelet

payload-processing-external-model-7489856dc7-h45cw

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x10)

models-as-a-service

kubelet

payload-processing-external-model-df4cf9975-hk75w

FailedMount

MountVolume.SetUp failed for volume "provider-credentials" : secret "agc-cred-8b97550813bf" not found

models-as-a-service

kubelet

payload-processing-external-model-7489856dc7-h45cw

Killing

Stopping container payload-processing

models-as-a-service

kubelet

payload-processing-external-model-df4cf9975-hk75w

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-extmod-a-6d170d-7cb75b8cb8-dkz4t

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-a-6d170d-5847588f6c-sp8v2

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-extmod-a-6d170d-openshift-default-86667cff57-m8zh9

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-extmod-b-951893-5d6556fd68-p6pfv

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-extmod-b-951893-openshift-default-586cb5d9bf-xmfj5

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-extmod-b-951893-6d666ffdd7-t8r4x

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

kubelet

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Killing

Stopping container payload-pre-processing

openshift-ingress

replicaset-controller

e2e-worker-w2-fba151-openshift-default-d5688d5cc

SuccessfulCreate

Created pod: e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4
(x2)

openshift-ingress

horizontal-pod-autoscaler

e2e-worker-w2-fba151-openshift-default

SuccessfulRescale

New size: 2; reason: cpu resource utilization (percentage of request) above target
(x2)

openshift-ingress

deployment-controller

e2e-worker-w2-fba151-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-worker-w2-fba151-openshift-default-d5688d5cc from 1 to 2

openshift-ingress

multus

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

AddedInterface

Add eth0 [10.134.0.78/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Created

Container created

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Started

Container started
(x7)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

ProbeError

Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused body:
(x7)

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Unhealthy

Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

ProbeError

Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) body:

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Unhealthy

Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
(x2)

openshift-ingress

kubelet

payload-pre-processing-e2e-worker-w2-fba151-6ccbb6bc67-j2n4b

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x2)

openshift-ingress

kubelet

payload-processing-e2e-worker-w2-fba151-59d7cc774-nw6l7

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-kzcq4

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-worker-w2-fba151-openshift-default-d5688d5cc-tg2bp

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503