=== events in openshift-ingress (2026-11-03T08:13:08Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 to ip-10-0-129-24.us-west-1.compute.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Successfully assigned openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Successfully assigned openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Successfully assigned openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg to ip-10-0-136-184.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Successfully assigned openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj to ip-10-0-129-24.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Successfully assigned openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 to ip-10-0-138-121.us-west-1.compute.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl to ip-10-0-138-121.us-west-1.compute.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g to ip-10-0-136-184.us-west-1.compute.internal 42m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd to ip-10-0-138-121.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j to ip-10-0-129-24.us-west-1.compute.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv to ip-10-0-136-184.us-west-1.compute.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 to ip-10-0-138-121.us-west-1.compute.internal 68m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-gvv6f Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Successfully assigned openshift-ingress/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 to ip-10-0-129-24.us-west-1.compute.internal 45m Normal Scheduled pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Successfully assigned openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw to ip-10-0-138-121.us-west-1.compute.internal 37m Normal Scheduled pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Successfully assigned openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47mcc to ip-10-0-129-24.us-west-1.compute.internal 65m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx to ip-10-0-138-121.us-west-1.compute.internal 46m Normal Scheduled pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Successfully assigned openshift-ingress/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Successfully assigned openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Successfully assigned openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z to ip-10-0-138-121.us-west-1.compute.internal 46m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrngs to ip-10-0-136-184.us-west-1.compute.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 to ip-10-0-138-121.us-west-1.compute.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 to ip-10-0-138-121.us-west-1.compute.internal 66m Normal Scheduled pod/kube-auth-proxy-86b9d545b-j69hh Successfully assigned openshift-ingress/kube-auth-proxy-86b9d545b-j69hh to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Successfully assigned openshift-ingress/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 to ip-10-0-136-184.us-west-1.compute.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd to ip-10-0-129-24.us-west-1.compute.internal 68m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp to ip-10-0-136-184.us-west-1.compute.internal 66m Normal Scheduled pod/kube-auth-proxy-86b9d545b-jsxrj Successfully assigned openshift-ingress/kube-auth-proxy-86b9d545b-jsxrj to ip-10-0-129-24.us-west-1.compute.internal 41m Normal Scheduled pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Successfully assigned openshift-ingress/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 to ip-10-0-129-24.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jgbz to ip-10-0-136-184.us-west-1.compute.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv to ip-10-0-129-24.us-west-1.compute.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 to ip-10-0-138-121.us-west-1.compute.internal 32m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5pm4h to ip-10-0-136-184.us-west-1.compute.internal 62m Normal Scheduled pod/payload-pre-processing-7b98bb678c-gxxn9 Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-gxxn9 to ip-10-0-129-24.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc to ip-10-0-138-121.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Successfully assigned openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m to ip-10-0-129-24.us-west-1.compute.internal 54m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh to ip-10-0-136-184.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz to ip-10-0-136-184.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Successfully assigned openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq to ip-10-0-129-24.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk to ip-10-0-129-24.us-west-1.compute.internal 56m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 to ip-10-0-129-24.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 to ip-10-0-138-121.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl to ip-10-0-136-184.us-west-1.compute.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg to ip-10-0-129-24.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7bv22 to ip-10-0-129-24.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Successfully assigned openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr to ip-10-0-138-121.us-west-1.compute.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v to ip-10-0-138-121.us-west-1.compute.internal 32m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8xt57 to ip-10-0-136-184.us-west-1.compute.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Successfully assigned openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f to ip-10-0-138-121.us-west-1.compute.internal 62m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr to ip-10-0-138-121.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 to ip-10-0-136-184.us-west-1.compute.internal 6m22s Normal Scheduled pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw to ip-10-0-138-121.us-west-1.compute.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 to ip-10-0-138-121.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb to ip-10-0-129-24.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx to ip-10-0-129-24.us-west-1.compute.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Successfully assigned openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s to ip-10-0-138-121.us-west-1.compute.internal 66m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 to ip-10-0-136-184.us-west-1.compute.internal 74m Normal Scheduled pod/router-default-699757bbb7-drfpj Successfully assigned openshift-ingress/router-default-699757bbb7-drfpj to ip-10-0-136-184.us-west-1.compute.internal 74m Warning FailedScheduling pod/router-default-699757bbb7-drfpj no nodes available to schedule pods 79m Warning FailedScheduling pod/router-default-699757bbb7-drfpj no nodes available to schedule pods 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv to ip-10-0-138-121.us-west-1.compute.internal 57m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr to ip-10-0-129-24.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 to ip-10-0-136-184.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb to ip-10-0-136-184.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n to ip-10-0-129-24.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Successfully assigned openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g to ip-10-0-136-184.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zvjvn to ip-10-0-136-184.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw to ip-10-0-138-121.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh to ip-10-0-129-24.us-west-1.compute.internal 41m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r to ip-10-0-138-121.us-west-1.compute.internal 68m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zlfl6 to ip-10-0-136-184.us-west-1.compute.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx to ip-10-0-138-121.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-xqv52 to ip-10-0-136-184.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Successfully assigned openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc to ip-10-0-138-121.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr to ip-10-0-136-184.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Successfully assigned openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r to ip-10-0-129-24.us-west-1.compute.internal 11m Normal Scheduled pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Successfully assigned openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd to ip-10-0-138-121.us-west-1.compute.internal 62m Normal Scheduled pod/payload-processing-55fbbbf9dc-hn2nn Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-hn2nn to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l to ip-10-0-129-24.us-west-1.compute.internal 16m Normal Scheduled pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Successfully assigned openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Successfully assigned openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq to ip-10-0-138-121.us-west-1.compute.internal 6m22s Normal Scheduled pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Successfully assigned openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 to ip-10-0-138-121.us-west-1.compute.internal 49m Normal Scheduled pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Successfully assigned openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 to ip-10-0-138-121.us-west-1.compute.internal 42m Normal Scheduled pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Successfully assigned openshift-ingress/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks to ip-10-0-136-184.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Successfully assigned openshift-ingress/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 to ip-10-0-129-24.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp to ip-10-0-129-24.us-west-1.compute.internal 40m Normal Scheduled pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Successfully assigned openshift-ingress/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k to ip-10-0-129-24.us-west-1.compute.internal 11m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw to ip-10-0-129-24.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 to ip-10-0-129-24.us-west-1.compute.internal 41m Normal Scheduled pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Successfully assigned openshift-ingress/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 to ip-10-0-129-24.us-west-1.compute.internal 15m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d to ip-10-0-129-24.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 to ip-10-0-138-121.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k to ip-10-0-136-184.us-west-1.compute.internal 41m Normal Scheduled pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Successfully assigned openshift-ingress/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 to ip-10-0-136-184.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 to ip-10-0-129-24.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 to ip-10-0-129-24.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Successfully assigned openshift-ingress/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf to ip-10-0-129-24.us-west-1.compute.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv to ip-10-0-138-121.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr to ip-10-0-136-184.us-west-1.compute.internal 33m Normal Scheduled pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Successfully assigned openshift-ingress/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp to ip-10-0-129-24.us-west-1.compute.internal 26m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg to ip-10-0-138-121.us-west-1.compute.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf to ip-10-0-138-121.us-west-1.compute.internal 57m Normal Scheduled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Successfully assigned openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n to ip-10-0-138-121.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Successfully assigned openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf to ip-10-0-138-121.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Successfully assigned openshift-ingress/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 to ip-10-0-129-24.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Successfully assigned openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g to ip-10-0-138-121.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Successfully assigned openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn to ip-10-0-138-121.us-west-1.compute.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b to ip-10-0-129-24.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr to ip-10-0-129-24.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z to ip-10-0-138-121.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml to ip-10-0-136-184.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt to ip-10-0-138-121.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Successfully assigned openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Successfully assigned openshift-ingress/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf to ip-10-0-129-24.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb to ip-10-0-136-184.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 to ip-10-0-129-24.us-west-1.compute.internal 11m Normal Scheduled pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm to ip-10-0-129-24.us-west-1.compute.internal 15m Normal Scheduled pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr to ip-10-0-129-24.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w to ip-10-0-136-184.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Successfully assigned openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf to ip-10-0-136-184.us-west-1.compute.internal 47m Normal Scheduled pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Successfully assigned openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd to ip-10-0-138-121.us-west-1.compute.internal 62m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 to ip-10-0-136-184.us-west-1.compute.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g to ip-10-0-138-121.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl to ip-10-0-138-121.us-west-1.compute.internal 41m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz to ip-10-0-129-24.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmkwr to ip-10-0-136-184.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv to ip-10-0-138-121.us-west-1.compute.internal 5m27s Normal Scheduled pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Successfully assigned openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x to ip-10-0-138-121.us-west-1.compute.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt to ip-10-0-129-24.us-west-1.compute.internal 23m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-rdzr6 to ip-10-0-136-184.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s to ip-10-0-138-121.us-west-1.compute.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn to ip-10-0-129-24.us-west-1.compute.internal 5m19s Normal Scheduled pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 to ip-10-0-129-24.us-west-1.compute.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 to ip-10-0-138-121.us-west-1.compute.internal 32m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qbnpn to ip-10-0-136-184.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-pmhtb to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 to ip-10-0-136-184.us-west-1.compute.internal 42m Normal Scheduled pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Successfully assigned openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd to ip-10-0-138-121.us-west-1.compute.internal 41m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk to ip-10-0-129-24.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-fvch8 to ip-10-0-136-184.us-west-1.compute.internal 32m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gbst7 to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth to ip-10-0-136-184.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf to ip-10-0-129-24.us-west-1.compute.internal 6m29s Normal Scheduled pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Successfully assigned openshift-ingress/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q to ip-10-0-129-24.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Successfully assigned openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 to ip-10-0-138-121.us-west-1.compute.internal 62m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-p5gnc to ip-10-0-129-24.us-west-1.compute.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 to ip-10-0-138-121.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mrk4m to ip-10-0-129-24.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kx6fw to ip-10-0-129-24.us-west-1.compute.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6 to ip-10-0-138-121.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl to ip-10-0-136-184.us-west-1.compute.internal 39m Normal Scheduled pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Successfully assigned openshift-ingress/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k to ip-10-0-129-24.us-west-1.compute.internal 54m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg to ip-10-0-136-184.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Successfully assigned openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Successfully assigned openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm to ip-10-0-138-121.us-west-1.compute.internal 33m Normal Scheduled pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Successfully assigned openshift-ingress/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h to ip-10-0-129-24.us-west-1.compute.internal 5m19s Normal Scheduled pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Successfully assigned openshift-ingress/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 to ip-10-0-129-24.us-west-1.compute.internal 47m Normal Scheduled pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Successfully assigned openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk to ip-10-0-138-121.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Successfully assigned openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks to ip-10-0-136-184.us-west-1.compute.internal 62m Normal Scheduled pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gmc69 to ip-10-0-129-24.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk to ip-10-0-138-121.us-west-1.compute.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs to ip-10-0-129-24.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Successfully assigned openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s to ip-10-0-138-121.us-west-1.compute.internal 79m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-699757bbb7 from 0 to 1 79m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 79m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 79m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 79m Normal SuccessfulCreate replicaset/router-default-699757bbb7 Created pod: router-default-699757bbb7-drfpj 74m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 74m Warning FailedMount pod/router-default-699757bbb7-drfpj MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 74m Warning FailedMount pod/router-default-699757bbb7-drfpj MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 74m Warning FailedMount pod/router-default-699757bbb7-drfpj MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 74m Warning FailedMount pod/router-default-699757bbb7-drfpj MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 73m Normal AddedInterface pod/router-default-699757bbb7-drfpj Add eth0 [10.132.0.20/23] from ovn-kubernetes 73m Normal Pulled pod/router-default-699757bbb7-drfpj Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 73m Normal Created pod/router-default-699757bbb7-drfpj Container created 73m Normal Started pod/router-default-699757bbb7-drfpj Container started 73m Warning ProbeError pod/router-default-699757bbb7-drfpj Startup probe error: HTTP probe failed with statuscode: 500... 73m Warning Unhealthy pod/router-default-699757bbb7-drfpj Startup probe failed: HTTP probe failed with statuscode: 500 68m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 68m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-gvv6f 68m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-gvv6f Add eth0 [10.133.0.22/23] from ovn-kubernetes 68m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-gvv6f Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 68m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 68m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-gvv6f Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.55s (4.55s including waiting). Image size: 185476477 bytes. 68m Normal Created pod/istiod-openshift-gateway-7876bfcb74-gvv6f Container created 68m Normal Started pod/istiod-openshift-gateway-7876bfcb74-gvv6f Container started 68m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-gvv6f Readiness probe error: HTTP probe failed with statuscode: 503... 68m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-gvv6f Readiness probe failed: HTTP probe failed with statuscode: 503 68m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 0 to 1 68m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-zlfl6 68m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-649845f967 from 0 to 1 68m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-649845f967 Created pod: openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp 68m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 68m Normal Pulling pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 68m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Add eth0 [10.132.0.31/23] from ovn-kubernetes 68m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 68m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Add eth0 [10.132.0.33/23] from ovn-kubernetes 68m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 68m Normal Created pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Container created 68m Normal Started pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Container started 68m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-649845f967-bmvkp Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.654s (9.654s including waiting). Image size: 183555333 bytes. 67m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 13.849s (13.849s including waiting). Image size: 183555333 bytes. 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 66m Normal SuccessfulCreate replicaset/kube-auth-proxy-86b9d545b Created pod: kube-auth-proxy-86b9d545b-j69hh 66m Normal SuccessfulCreate replicaset/kube-auth-proxy-86b9d545b Created pod: kube-auth-proxy-86b9d545b-jsxrj 66m Normal AddedInterface pod/kube-auth-proxy-86b9d545b-jsxrj Add eth0 [10.134.0.24/23] from ovn-kubernetes 66m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-69f549cf5c Created pod: data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 66m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-86b9d545b from 0 to 2 66m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-69f549cf5c from 0 to 1 66m Normal AddedInterface pod/kube-auth-proxy-86b9d545b-j69hh Add eth0 [10.133.0.24/23] from ovn-kubernetes 66m Normal Started pod/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 Container started 66m Normal Created pod/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 Container created 66m Normal Pulled pod/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 66m Normal Pulling pod/kube-auth-proxy-86b9d545b-jsxrj Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 66m Normal Pulling pod/kube-auth-proxy-86b9d545b-j69hh Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 66m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-69f549cf5c7nkh9 Add eth0 [10.132.0.41/23] from ovn-kubernetes 66m Normal Pulled pod/kube-auth-proxy-86b9d545b-j69hh Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.892s (2.892s including waiting). Image size: 188904677 bytes. 66m Normal Pulled pod/kube-auth-proxy-86b9d545b-jsxrj Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.917s (2.917s including waiting). Image size: 188904677 bytes. 66m Normal Created pod/kube-auth-proxy-86b9d545b-j69hh Container created 66m Normal Started pod/kube-auth-proxy-86b9d545b-j69hh Container started 66m Normal Created pod/kube-auth-proxy-86b9d545b-jsxrj Container created 66m Normal Started pod/kube-auth-proxy-86b9d545b-jsxrj Container started 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 66m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 66m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 65m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-c8rtx 65m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Add eth0 [10.133.0.31/23] from ovn-kubernetes 65m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 65m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Container created 65m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Container started 65m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Readiness probe failed: HTTP probe failed with statuscode: 503 65m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx Readiness probe error: HTTP probe failed with statuscode: 503... 65m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-gvv6f 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 65m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-gvv6f Stopping container discovery 65m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 65m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Container started 65m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Container created 65m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Startup probe error: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused... 65m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Startup probe failed: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused 64m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 62m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 1 to 3 62m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-p5gnc 62m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-x59tm 62m Normal Pulling pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 62m Normal Pulling pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 62m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Add eth0 [10.133.0.40/23] from ovn-kubernetes 62m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Add eth0 [10.134.0.31/23] from ovn-kubernetes 62m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Container created 62m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Container started 62m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.724s (3.724s including waiting). Image size: 183555333 bytes. 62m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Container started 62m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Container created 62m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.516s (3.516s including waiting). Image size: 183555333 bytes. 62m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Startup probe error: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused... 62m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Startup probe failed: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused 62m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 62m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 62m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1 62m Warning FailedCreate replicaset/payload-processing-55fbbbf9dc Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 62m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 62m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-gxxn9 62m Normal Pulling pod/payload-pre-processing-7b98bb678c-gxxn9 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-gxxn9 Add eth0 [10.134.0.33/23] from ovn-kubernetes 62m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 62m Normal SuccessfulCreate replicaset/payload-processing-55fbbbf9dc Created pod: payload-processing-55fbbbf9dc-hn2nn 62m Normal AddedInterface pod/payload-processing-55fbbbf9dc-hn2nn Add eth0 [10.133.0.42/23] from ovn-kubernetes 62m Normal Pulling pod/payload-processing-55fbbbf9dc-hn2nn Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal Started pod/payload-processing-55fbbbf9dc-hn2nn Container started 62m Normal Started pod/payload-pre-processing-7b98bb678c-gxxn9 Container started 62m Normal Pulled pod/payload-pre-processing-7b98bb678c-gxxn9 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.735s (2.735s including waiting). Image size: 125353963 bytes. 62m Normal Pulled pod/payload-processing-55fbbbf9dc-hn2nn Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.495s (2.495s including waiting). Image size: 125353963 bytes. 62m Normal Created pod/payload-pre-processing-7b98bb678c-gxxn9 Container created 62m Normal Created pod/payload-processing-55fbbbf9dc-hn2nn Container created 62m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 62m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 3 to 5 62m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Add eth0 [10.134.0.35/23] from ovn-kubernetes 62m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Add eth0 [10.133.0.43/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-dk9tr 62m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-gmc69 62m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Container started 62m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Container created 62m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Container created 62m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Container started 62m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused 62m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Startup probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused... 62m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Startup probe failed: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused 62m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Startup probe error: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused... 61m Normal AddedLabel gateway/e2e-worker-w3-1f2e7e Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-1f2e7e 61m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p 61m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 0 to 1 61m Normal AddedLabel gateway/e2e-worker-w6-3b41b9 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-3b41b9 61m Normal AddedLabel gateway/e2e-worker-w1-9697f7 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-9697f7 61m Normal AddedLabel gateway/e2e-worker-w0-6606d6 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-6606d6 61m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled up replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 0 to 1 61m Normal AddedLabel gateway/e2e-worker-w4-ffee69 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-ffee69 61m Normal AddedLabel gateway/e2e-worker-w5-3fea3a Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-3fea3a 61m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g 61m Normal AddedLabel gateway/e2e-worker-w2-669d6b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-669d6b 61m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 0 to 1 61m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv 61m Normal AddedInterface pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Add eth0 [10.134.0.37/23] from ovn-kubernetes 61m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Add eth0 [10.133.0.44/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled up replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 0 to 1 61m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Add eth0 [10.134.0.36/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Created pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j 61m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 61m Normal ScalingReplicaSet deployment/e2e-worker-w6-3b41b9-openshift-default Scaled up replica set e2e-worker-w6-3b41b9-openshift-default-84578dcc5f from 0 to 1 61m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Add eth0 [10.133.0.45/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s 61m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Add eth0 [10.134.0.38/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 61m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Add eth0 [10.133.0.46/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 0 to 1 61m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 0 to 1 61m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Add eth0 [10.134.0.39/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-9697f7 Scaled up replica set payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c from 0 to 1 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-9697f7 Scaled up replica set payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455 from 0 to 1 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Add eth0 [10.133.0.47/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455 Created pod: payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455 Error creating: pods "payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-9697f7: serviceaccount "payload-processing-e2e-worker-w1-9697f7" not found 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c Created pod: payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c Error creating: pods "payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-9697f7: serviceaccount "payload-processing-e2e-worker-w1-9697f7" not found 61m Normal AddedInterface pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Add eth0 [10.133.0.48/23] from ovn-kubernetes 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Container started 61m Normal Created pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Container created 61m Normal Pulled pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Container started 61m Normal Created pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Container created 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-669d6b Scaled up replica set payload-processing-e2e-worker-w2-669d6b-86d46969f7 from 0 to 1 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-669d6b Scaled up replica set payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-669d6b-86d46969f7 Created pod: payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc Error creating: pods "payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-669d6b: serviceaccount "payload-processing-e2e-worker-w2-669d6b" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc Created pod: payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-669d6b-86d46969f7 Error creating: pods "payload-processing-e2e-worker-w2-669d6b-86d46969f7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-669d6b: serviceaccount "payload-processing-e2e-worker-w2-669d6b" not found 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Created pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Container created 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Add eth0 [10.134.0.40/23] from ovn-kubernetes 61m Normal Started pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Container started 61m Normal Created pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Container created 61m Normal AddedInterface pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Add eth0 [10.134.0.41/23] from ovn-kubernetes 61m Normal Pulled pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Container started 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785 Error creating: pods "payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e: serviceaccount "payload-processing-e2e-worker-w3-1f2e7e" not found 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-1f2e7e Scaled up replica set payload-processing-e2e-worker-w3-1f2e7e-6cb6659785 from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f Created pod: payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785 Created pod: payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f Error creating: pods "payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-1f2e7e: serviceaccount "payload-processing-e2e-worker-w3-1f2e7e" not found 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-1f2e7e Scaled up replica set payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f from 0 to 1 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Add eth0 [10.134.0.42/23] from ovn-kubernetes 61m Normal Pulled pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Created pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Container created 61m Normal Started pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Container started 61m Normal AddedInterface pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Add eth0 [10.134.0.43/23] from ovn-kubernetes 61m Normal Started pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Container started 61m Normal Created pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Container created 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-6606d6-868d47b4ff Error creating: pods "payload-processing-e2e-worker-w0-6606d6-868d47b4ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6606d6: serviceaccount "payload-processing-e2e-worker-w0-6606d6" not found 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-6606d6-868d47b4ff Created pod: payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867 Created pod: payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867 Error creating: pods "payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6606d6: serviceaccount "payload-processing-e2e-worker-w0-6606d6" not found 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-6606d6 Scaled up replica set payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867 from 0 to 1 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-6606d6 Scaled up replica set payload-processing-e2e-worker-w0-6606d6-868d47b4ff from 0 to 1 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Container started 61m Normal Pulled pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Container started 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Add eth0 [10.133.0.51/23] from ovn-kubernetes 61m Normal Created pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Container created 61m Normal Created pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Container created 61m Normal AddedInterface pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Add eth0 [10.134.0.44/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-ffee69 Scaled up replica set payload-pre-processing-e2e-worker-w4-ffee69-757d59959f from 0 to 1 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f Error creating: pods "payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-ffee69: serviceaccount "payload-processing-e2e-worker-w4-ffee69" not found 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf Error creating: pods "payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-ffee69: serviceaccount "payload-processing-e2e-worker-w4-ffee69" not found 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf Created pod: payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f Created pod: payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-ffee69 Scaled up replica set payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf from 0 to 1 61m Normal Pulled pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Add eth0 [10.133.0.52/23] from ovn-kubernetes 61m Normal AddedInterface pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Add eth0 [10.134.0.45/23] from ovn-kubernetes 61m Normal Created pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Container created 61m Normal Started pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Container started 61m Normal Created pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Container created 61m Normal Started pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Container started 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-3fea3a Scaled up replica set payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87 from 0 to 1 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87 Error creating: pods "payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-3fea3a: serviceaccount "payload-processing-e2e-worker-w5-3fea3a" not found 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-3fea3a Scaled up replica set payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7 from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87 Created pod: payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7 Created pod: payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7 Error creating: pods "payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-3fea3a: serviceaccount "payload-processing-e2e-worker-w5-3fea3a" not found 61m Normal Pulled pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal Created pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Container created 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Add eth0 [10.134.0.46/23] from ovn-kubernetes 61m Normal Created pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Container created 61m Normal AddedInterface pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Add eth0 [10.134.0.47/23] from ovn-kubernetes 61m Normal Started pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Container started 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal Started pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Container started 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-3b41b9-597b56455c Error creating: pods "payload-processing-e2e-worker-w6-3b41b9-597b56455c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-3b41b9: serviceaccount "payload-processing-e2e-worker-w6-3b41b9" not found 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-3b41b9 Scaled up replica set payload-processing-e2e-worker-w6-3b41b9-597b56455c from 0 to 1 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-3b41b9 Scaled up replica set payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-3b41b9-597b56455c Created pod: payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c Error creating: pods "payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-3b41b9: serviceaccount "payload-processing-e2e-worker-w6-3b41b9" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c Created pod: payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 61m Normal AddedInterface pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Add eth0 [10.133.0.53/23] from ovn-kubernetes 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Add eth0 [10.134.0.48/23] from ovn-kubernetes 61m Normal Created pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Container created 61m Normal Started pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Container started 61m Normal Created pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Container created 61m Normal Started pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Container started 61m Normal Pulled pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Container started 61m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Container created 61m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Container created 61m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Container started 61m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 61m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 61m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Container started 61m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Container created 60m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused... 60m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Container created 60m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Container started 60m Normal Pulled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Created pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Container created 60m Normal Started pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Container started 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused... 60m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused 60m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Container created 60m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Container started 60m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Container started 60m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Container created 60m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Startup probe error: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n 57m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Container created 57m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Add eth0 [10.133.0.59/23] from ovn-kubernetes 57m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Container started 57m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 57m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 5 to 4 57m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-x59tm 57m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-x59tm Stopping container istio-proxy 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr 57m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 1 to 2 57m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Add eth0 [10.134.0.65/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Container started 57m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Container created 56m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled up replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 1 to 2 56m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Add eth0 [10.134.0.67/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Container started 56m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Container created 56m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 56m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-p5gnc 56m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-p5gnc Stopping container istio-proxy 56m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 4 to 3 56m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 55m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-dk9tr 55m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr Stopping container istio-proxy 55m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 55m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 3 to 2 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m 55m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 2 to 3 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Created pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled up replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 2 to 3 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled up replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 1 to 2 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 1 to 2 55m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 1 to 2 55m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 2 to 4 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-3b41b9-openshift-default Scaled up replica set e2e-worker-w6-3b41b9-openshift-default-84578dcc5f from 1 to 2 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 55m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Add eth0 [10.132.0.63/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Container created 55m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Container started 55m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Container started 55m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Container created 55m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Add eth0 [10.133.0.63/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Container created 55m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Add eth0 [10.133.0.64/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf 55m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Container started 55m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Add eth0 [10.132.0.64/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Container created 55m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Container started 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 55m Normal AddedInterface pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Add eth0 [10.133.0.62/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Container created 55m Normal Started pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Container started 55m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Container started 55m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Add eth0 [10.134.0.81/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Container created 55m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Add eth0 [10.133.0.61/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Container started 55m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Container created 55m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Container created 55m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Container started 55m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Add eth0 [10.132.0.65/23] from ovn-kubernetes 55m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Startup probe error: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Startup probe failed: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Startup probe error: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Startup probe failed: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Startup probe failed: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Startup probe error: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Startup probe error: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Startup probe failed: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 2 to 3 54m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg 54m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Container started 54m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Add eth0 [10.132.0.68/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Container created 54m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled up replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 2 to 3 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh 54m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Add eth0 [10.132.0.69/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Container started 54m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Container created 53m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 52m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-gmc69 52m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 1; reason: All metrics below target 52m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 2 to 1 52m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Stopping container istio-proxy 52m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Readiness probe failed: HTTP probe failed with statuscode: 503 52m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-gmc69 Readiness probe error: HTTP probe failed with statuscode: 503... 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 3 to 5 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg 52m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Container started 52m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Container created 52m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Add eth0 [10.134.0.91/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Container created 52m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Add eth0 [10.133.0.67/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Container started 52m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 5 to 6 52m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Container created 52m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 2 to 3 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 52m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Add eth0 [10.132.0.71/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf 52m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Container started 52m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Container created 52m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Add eth0 [10.132.0.72/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Container started 52m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled up replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 3 to 6 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg 51m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled up replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 3 to 6 51m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 6 to 10 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 4 to 8 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc 51m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Add eth0 [10.133.0.72/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Container created 51m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Container started 51m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Add eth0 [10.134.0.102/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Add eth0 [10.134.0.101/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Add eth0 [10.133.0.71/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Container created 51m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Container created 51m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Container started 51m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Add eth0 [10.132.0.76/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Add eth0 [10.132.0.73/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Add eth0 [10.134.0.99/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Container created 51m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Container started 51m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Add eth0 [10.134.0.100/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Container created 51m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Container started 51m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Container created 51m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Add eth0 [10.132.0.74/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 51m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Add eth0 [10.133.0.70/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Container created 51m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Container started 51m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr 51m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Add eth0 [10.132.0.75/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Container started 51m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Container created 51m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc (combined from similar events): Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 51m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Container created 51m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Add eth0 [10.134.0.98/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Add eth0 [10.133.0.73/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Container created 51m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Container started 51m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Container started 51m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Add eth0 [10.133.0.74/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Container started 51m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Container created 51m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Container started 51m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Container created 51m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Container started 51m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Container created 51m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Container created 51m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Container started 51m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Readiness probe error: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Readiness probe failed: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Readiness probe failed: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Readiness probe error: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Readiness probe failed: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Readiness probe error: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Readiness probe failed: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Readiness probe error: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Startup probe failed: Get "http://10.134.0.98:15021/healthz/ready": dial tcp 10.134.0.98:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Startup probe error: Get "http://10.134.0.98:15021/healthz/ready": dial tcp 10.134.0.98:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Startup probe failed: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Startup probe error: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Startup probe failed: Get "http://10.134.0.102:15021/healthz/ready": dial tcp 10.134.0.102:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Startup probe failed: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Startup probe error: Get "http://10.134.0.102:15021/healthz/ready": dial tcp 10.134.0.102:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Startup probe error: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Startup probe failed: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Startup probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Startup probe failed: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Startup probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Startup probe failed: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Startup probe error: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Startup probe error: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Startup probe error: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 9; reason: 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 8 to 9 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Add eth0 [10.132.0.77/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Container started 51m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Container created 51m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 3 to 4 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr 51m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Add eth0 [10.134.0.108/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Container created 51m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Container started 51m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Startup probe error: Get "http://10.134.0.108:15021/healthz/ready": dial tcp 10.134.0.108:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Startup probe failed: Get "http://10.134.0.108:15021/healthz/ready": dial tcp 10.134.0.108:15021: connect: connection refused 50m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 3 to 6 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z 50m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Add eth0 [10.132.0.78/23] from ovn-kubernetes 50m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Add eth0 [10.133.0.75/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Container started 50m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Container created 50m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Container started 50m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Container created 50m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Container started 50m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Container created 50m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Add eth0 [10.134.0.119/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Startup probe failed: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Startup probe error: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Startup probe error: Get "http://10.134.0.119:15021/healthz/ready": dial tcp 10.134.0.119:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Startup probe failed: Get "http://10.134.0.119:15021/healthz/ready": dial tcp 10.134.0.119:15021: connect: connection refused 50m Normal AddedLabel gateway/e2e-ait-67db4bc0 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-67db4bc0 50m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 6 to 8 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 8; reason: 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr 50m Normal AddedLabel gateway/e2e-migrate-33a46b37-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-33a46b37-gw 50m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Container started 50m Normal SuccessfulCreate replicaset/e2e-ait-67db4bc0-openshift-default-6b845d74fc Created pod: e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd 50m Normal ScalingReplicaSet deployment/e2e-ait-67db4bc0-openshift-default Scaled up replica set e2e-ait-67db4bc0-openshift-default-6b845d74fc from 0 to 1 50m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Container created 50m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Add eth0 [10.134.0.127/23] from ovn-kubernetes 50m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Add eth0 [10.133.0.76/23] from ovn-kubernetes 50m Normal AddedInterface pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Add eth0 [10.134.0.129/23] from ovn-kubernetes 50m Normal AddedInterface pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Add eth0 [10.133.0.77/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9 Created pod: e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 50m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Container started 50m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Container created 50m Normal ScalingReplicaSet deployment/e2e-migrate-33a46b37-gw-openshift-default Scaled up replica set e2e-migrate-33a46b37-gw-openshift-default-84976c56d9 from 0 to 1 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Startup probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Startup probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Startup probe failed: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Startup probe error: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused... 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-33a46b37 Scaled up replica set payload-processing-e2e-migrate-33a46b37-5d8c64fb59 from 0 to 1 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-33a46b37 Scaled up replica set payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc from 0 to 1 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc Error creating: pods "payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-33a46b37: serviceaccount "payload-processing-e2e-migrate-33a46b37" not found 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc Created pod: payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf 50m Warning FailedCreate replicaset/payload-processing-e2e-migrate-33a46b37-5d8c64fb59 Error creating: pods "payload-processing-e2e-migrate-33a46b37-5d8c64fb59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-33a46b37: serviceaccount "payload-processing-e2e-migrate-33a46b37" not found 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-33a46b37-5d8c64fb59 Created pod: payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml 50m Normal AddedInterface pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Add eth0 [10.133.0.79/23] from ovn-kubernetes 50m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Add eth0 [10.133.0.78/23] from ovn-kubernetes 50m Normal Pulled pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Container created 50m Normal Pulled pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Started pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Container started 50m Normal Created pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Container created 50m Normal Started pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Container started 50m Warning FailedCreate replicaset/payload-processing-e2e-ait-67db4bc0-5586ddb8c5 Error creating: pods "payload-processing-e2e-ait-67db4bc0-5586ddb8c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-67db4bc0: serviceaccount "payload-processing-e2e-ait-67db4bc0" not found 50m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c (combined from similar events): Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m 50m Normal ScalingReplicaSet deployment/e2e-worker-w6-3b41b9-openshift-default Scaled up replica set e2e-worker-w6-3b41b9-openshift-default-84578dcc5f from 2 to 4 50m Normal SuccessfulCreate replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Created pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj 50m Normal SuccessfulCreate replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Created pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 4 to 8 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-67db4bc0 Scaled up replica set payload-pre-processing-e2e-ait-67db4bc0-74d658899f from 0 to 1 50m Normal SuccessfulCreate replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 (combined from similar events): Created pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-67db4bc0-74d658899f Error creating: pods "payload-pre-processing-e2e-ait-67db4bc0-74d658899f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-67db4bc0: serviceaccount "payload-processing-e2e-ait-67db4bc0" not found 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled up replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 9 to 10 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-67db4bc0 Scaled up replica set payload-processing-e2e-ait-67db4bc0-5586ddb8c5 from 0 to 1 50m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc 50m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 50m Normal SuccessfulCreate replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Created pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled up replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 6 to 10 50m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Add eth0 [10.134.0.135/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Container started 50m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Container created 50m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Add eth0 [10.132.0.81/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Container created 50m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Container started 50m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Add eth0 [10.133.0.82/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Add eth0 [10.132.0.79/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Add eth0 [10.133.0.83/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Container started 50m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Container created 50m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Add eth0 [10.132.0.80/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-67db4bc0-5586ddb8c5 Created pod: payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 50m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Add eth0 [10.134.0.133/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Add eth0 [10.133.0.80/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Container created 50m Normal Started pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Container started 50m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Container created 50m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Container started 50m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Container started 50m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Container created 50m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Add eth0 [10.134.0.134/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Container created 50m Normal Started pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Container started 50m Normal Created pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Container created 50m Normal Started pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Container started 50m Normal AddedInterface pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Add eth0 [10.134.0.132/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-67db4bc0-74d658899f Created pod: payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 50m Normal Pulled pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Container created 50m Normal Started pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Container started 50m Normal AddedInterface pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Add eth0 [10.133.0.81/23] from ovn-kubernetes 50m Normal Created pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Container created 50m Normal Pulled pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Started pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Container started 50m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Container started 50m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Container created 50m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Container created 50m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Container created 50m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Container started 50m Normal Started pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Container started 50m Normal AddedInterface pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Add eth0 [10.134.0.136/23] from ovn-kubernetes 50m Normal AddedInterface pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Add eth0 [10.133.0.84/23] from ovn-kubernetes 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 50m Normal AddedLabel gateway/e2e-ait-5c169f6a Added label istio.io/rev=openshift-gateway to gateway e2e-ait-5c169f6a 50m Normal SuccessfulCreate replicaset/e2e-ait-5c169f6a-openshift-default-5d598cd459 Created pod: e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Startup probe failed: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused 50m Normal ScalingReplicaSet deployment/e2e-ait-5c169f6a-openshift-default Scaled up replica set e2e-ait-5c169f6a-openshift-default-5d598cd459 from 0 to 1 50m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Startup probe error: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Startup probe error: Get "http://10.134.0.133:15021/healthz/ready": dial tcp 10.134.0.133:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Readiness probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Startup probe error: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Startup probe failed: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Startup probe failed: Get "http://10.134.0.133:15021/healthz/ready": dial tcp 10.134.0.133:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Readiness probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Startup probe failed: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Readiness probe error: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Readiness probe failed: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Readiness probe failed: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Startup probe error: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Readiness probe error: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Readiness probe error: Get "http://10.133.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Readiness probe error: Get "http://10.133.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Startup probe error: Get "http://10.134.0.132:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Startup probe failed: Get "http://10.134.0.132:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Startup probe failed: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Readiness probe error: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Readiness probe error: Get "http://10.134.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Readiness probe failed: Get "http://10.134.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-47mcc 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Readiness probe failed: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Readiness probe error: Get "http://10.134.0.99:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 50m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 50m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 9; reason: All metrics below target 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Readiness probe failed: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Startup probe failed: Get "http://10.134.0.134:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled up replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 8 to 9 50m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Startup probe failed: Get "http://10.134.0.133:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Startup probe failed: Get "http://10.133.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Created pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks 50m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Startup probe error: Get "http://10.134.0.133:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Readiness probe error: Get "http://10.134.0.102:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Readiness probe failed: Get "http://10.134.0.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Readiness probe error: Get "http://10.134.0.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Startup probe error: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Readiness probe failed: Get "http://10.133.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Readiness probe error: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Readiness probe error: Get "http://10.133.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Normal AddedInterface pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Add eth0 [10.133.0.85/23] from ovn-kubernetes 49m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Startup probe error: Get "http://10.133.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Startup probe error: Get "http://10.134.0.134:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Readiness probe failed: timeout: health rpc did not complete within 1s 49m Normal Created pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Container created 49m Normal AddedInterface pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Add eth0 [10.132.0.82/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Container started 49m Normal Pulled pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 49m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Add eth0 [10.134.0.138/23] from ovn-kubernetes 49m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Container started 49m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Container created 49m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.931s (3.931s including waiting). Image size: 185476477 bytes. 49m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 49m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Readiness probe failed: HTTP probe failed with statuscode: 503 49m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Readiness probe error: HTTP probe failed with statuscode: 503... 49m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-5c169f6a Scaled up replica set payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847 from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-ait-5c169f6a-6769b6f497 Error creating: pods "payload-processing-e2e-ait-5c169f6a-6769b6f497-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-5c169f6a: serviceaccount "payload-processing-e2e-ait-5c169f6a" not found 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847 Error creating: pods "payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-5c169f6a: serviceaccount "payload-processing-e2e-ait-5c169f6a" not found 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-5c169f6a Scaled up replica set payload-processing-e2e-ait-5c169f6a-6769b6f497 from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847 Created pod: payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-5c169f6a-6769b6f497 Created pod: payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 49m Normal Created pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Container created 49m Normal Pulled pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Add eth0 [10.133.0.86/23] from ovn-kubernetes 49m Normal Pulled pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Container created 49m Normal Started pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Container started 49m Normal AddedInterface pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Add eth0 [10.133.0.87/23] from ovn-kubernetes 49m Normal Started pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Container started 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-kx6fw 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-zvjvn 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-s4f8g 49m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 1 to 4 49m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Container created 49m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Container started 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Add eth0 [10.133.0.89/23] from ovn-kubernetes 49m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Add eth0 [10.132.0.83/23] from ovn-kubernetes 49m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Container created 49m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Container started 49m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Container created 49m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Container started 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Add eth0 [10.134.0.139/23] from ovn-kubernetes 49m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx 49m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Add eth0 [10.132.0.84/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c (combined from similar events): Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Created pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled up replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 8 to 10 49m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 49m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Add eth0 [10.133.0.91/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw 49m Normal SuccessfulCreate replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 (combined from similar events): Created pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled up replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 6 to 10 49m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Add eth0 [10.134.0.140/23] from ovn-kubernetes 49m Normal AddedInterface pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Add eth0 [10.132.0.85/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Container started 49m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Container created 49m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Container started 49m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Container started 49m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Container started 49m Normal Started pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Container started 49m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Container created 49m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Add eth0 [10.132.0.86/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Container created 49m Normal Pulled pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Container created 49m Normal Pulled pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Add eth0 [10.132.0.87/23] from ovn-kubernetes 49m Normal Created pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Container created 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Startup probe failed: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused 49m Normal Created pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Container created 49m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Startup probe error: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Startup probe failed: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused 49m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Startup probe error: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused... 49m Normal Started pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Container started 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 49m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Startup probe failed: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Startup probe error: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Readiness probe failed: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Readiness probe error: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal Pulled pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Container started 49m Normal Created pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Container created 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-67db4bc0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-67db4bc0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning ProbeError pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Startup probe error: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Startup probe failed: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-pmhtb 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-qrckb 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-tmkwr 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-gp9f6 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Container created 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Container started 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Add eth0 [10.134.0.141/23] from ovn-kubernetes 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Container started 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Container created 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Container started 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Container created 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Add eth0 [10.133.0.93/23] from ovn-kubernetes 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Add eth0 [10.132.0.88/23] from ovn-kubernetes 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Add eth0 [10.133.0.94/23] from ovn-kubernetes 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Container started 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Container created 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Startup probe error: Get "http://10.134.0.141:15021/healthz/ready": dial tcp 10.134.0.141:15021: connect: connection refused... 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Startup probe failed: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Startup probe failed: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Startup probe error: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused... 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Startup probe failed: Get "http://10.134.0.141:15021/healthz/ready": dial tcp 10.134.0.141:15021: connect: connection refused 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Startup probe error: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused... 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-fvch8 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-mrk4m 48m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default (combined from similar events): Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 8 to 10 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Add eth0 [10.134.0.142/23] from ovn-kubernetes 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Container started 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Add eth0 [10.132.0.89/23] from ovn-kubernetes 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Container created 48m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Container created 48m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Container started 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-67db4bc0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-67db4bc0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused... 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 48m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 48m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused 48m Warning ProbeError pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Readiness probe error: Get "http://10.134.0.129:15021/healthz/ready": read tcp 10.134.0.2:44694->10.134.0.129:15021: read: connection reset by peer... 48m Warning Unhealthy pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Readiness probe failed: Get "http://10.134.0.129:15021/healthz/ready": read tcp 10.134.0.2:44694->10.134.0.129:15021: read: connection reset by peer 48m Normal Pulled pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning Unhealthy pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Readiness probe failed: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Normal Created pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Container created 48m Warning ProbeError pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Readiness probe error: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Normal Started pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Container started 48m Normal Pulled pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Container started 48m Normal Created pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Container created 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-5c169f6a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-5c169f6a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning Unhealthy pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Startup probe failed: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused 48m Warning ProbeError pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Startup probe error: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Startup probe failed: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused 48m Warning ProbeError pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Startup probe error: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused... 48m Normal Killing pod/payload-processing-e2e-ait-67db4bc0-5586ddb8c5-dpck9 Stopping container payload-processing 48m Normal Killing pod/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2 Stopping container payload-pre-processing 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-33a46b37-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-33a46b37-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Killing pod/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml Stopping container payload-processing 48m Normal Killing pod/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf Stopping container payload-pre-processing 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-33a46b37-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-33a46b37-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-5c169f6a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-5c169f6a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Killing pod/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8 Stopping container payload-processing 47m Normal Killing pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd Stopping container istio-proxy 47m Normal ScalingReplicaSet deployment/e2e-ait-6d089af5-openshift-default Scaled up replica set e2e-ait-6d089af5-openshift-default-66c5f9b66c from 0 to 1 47m Normal AddedLabel gateway/e2e-ait-6d089af5 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-6d089af5 47m Normal SuccessfulCreate replicaset/e2e-ait-6d089af5-openshift-default-66c5f9b66c Created pod: e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr 47m Normal Killing pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Stopping container istio-proxy 47m Normal AddedInterface pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Add eth0 [10.133.0.95/23] from ovn-kubernetes 47m Normal AddedLabel gateway/e2e-derive-209c4c70 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-209c4c70 47m Normal ScalingReplicaSet deployment/e2e-derive-209c4c70-openshift-default Scaled up replica set e2e-derive-209c4c70-openshift-default-778f9c776f from 0 to 1 47m Normal SuccessfulCreate replicaset/e2e-derive-209c4c70-openshift-default-778f9c776f Created pod: e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl 47m Warning Unhealthy pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Readiness probe failed: HTTP probe failed with statuscode: 503 47m Normal AddedInterface pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Add eth0 [10.133.0.96/23] from ovn-kubernetes 47m Warning ProbeError pod/e2e-migrate-33a46b37-gw-openshift-default-84976c56d9-55cp8 Readiness probe error: HTTP probe failed with statuscode: 503... 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-6d089af5 Scaled up replica set payload-pre-processing-e2e-ait-6d089af5-c489f7775 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-6d089af5 Scaled up replica set payload-processing-e2e-ait-6d089af5-6764877fbf from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-ait-6d089af5-6764877fbf Error creating: pods "payload-processing-e2e-ait-6d089af5-6764877fbf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-6d089af5: serviceaccount "payload-processing-e2e-ait-6d089af5" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-6d089af5-6764877fbf Created pod: payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-6d089af5-c489f7775 Error creating: pods "payload-pre-processing-e2e-ait-6d089af5-c489f7775-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-6d089af5: serviceaccount "payload-processing-e2e-ait-6d089af5" not found 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-6d089af5-c489f7775 Created pod: payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx 47m Normal Created pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Container created 47m Normal Started pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Container started 47m Normal Pulled pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Container created 47m Normal Started pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Container started 47m Normal AddedInterface pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Add eth0 [10.133.0.98/23] from ovn-kubernetes 47m Normal AddedInterface pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Add eth0 [10.133.0.97/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-209c4c70 Scaled up replica set payload-processing-e2e-derive-209c4c70-547cdcf4dc from 0 to 1 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc Error creating: pods "payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-209c4c70: serviceaccount "payload-processing-e2e-derive-209c4c70" not found 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-209c4c70 Scaled up replica set payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc Created pod: payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-209c4c70-547cdcf4dc Created pod: payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk 47m Warning FailedCreate replicaset/payload-processing-e2e-derive-209c4c70-547cdcf4dc Error creating: pods "payload-processing-e2e-derive-209c4c70-547cdcf4dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-209c4c70: serviceaccount "payload-processing-e2e-derive-209c4c70" not found 47m Normal AddedInterface pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Add eth0 [10.133.0.100/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Add eth0 [10.133.0.99/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Container created 47m Normal Started pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Container started 47m Normal Created pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Container created 47m Normal Started pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Container started 47m Normal Killing pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Stopping container istio-proxy 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-209c4c70-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-209c4c70-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning ProbeError pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Readiness probe error: HTTP probe failed with statuscode: 503... 47m Warning Unhealthy pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps Readiness probe failed: HTTP probe failed with statuscode: 503 46m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-wrngs 46m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Add eth0 [10.132.0.90/23] from ovn-kubernetes 46m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 46m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 46m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 46m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Container started 46m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.992s (3.992s including waiting). Image size: 185476477 bytes. 46m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Container created 46m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Readiness probe failed: HTTP probe failed with statuscode: 503 46m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Readiness probe error: HTTP probe failed with statuscode: 503... 46m Normal AddedLabel gateway/e2e-shared-a-w5-893d83 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-893d83 46m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc Created pod: e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf 46m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-893d83-openshift-default Scaled up replica set e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc from 0 to 1 46m Normal AddedInterface pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Add eth0 [10.133.0.102/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/e2e-mt-399d09e5-openshift-default-85d5f99789 Created pod: e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-893d83 Scaled up replica set payload-processing-e2e-shared-a-w5-893d83-b64d457d5 from 0 to 1 46m Normal ScalingReplicaSet deployment/e2e-mt-399d09e5-openshift-default Scaled up replica set e2e-mt-399d09e5-openshift-default-85d5f99789 from 0 to 1 46m Normal AddedLabel gateway/e2e-mt-399d09e5 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-399d09e5 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-893d83 Scaled up replica set payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5 from 0 to 1 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5 Error creating: pods "payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-893d83: serviceaccount "payload-processing-e2e-shared-a-w5-893d83" not found 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-893d83-b64d457d5 Error creating: pods "payload-processing-e2e-shared-a-w5-893d83-b64d457d5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-893d83: serviceaccount "payload-processing-e2e-shared-a-w5-893d83" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5 Created pod: payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Add eth0 [10.134.0.148/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-893d83-b64d457d5 Created pod: payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv 46m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Container started 46m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Container created 46m Normal AddedInterface pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Add eth0 [10.133.0.104/23] from ovn-kubernetes 46m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Add eth0 [10.134.0.149/23] from ovn-kubernetes 46m Normal Started pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Container started 46m Normal Created pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Container created 46m Normal Pulled pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-399d09e5 Scaled up replica set payload-pre-processing-e2e-mt-399d09e5-6d9766775f from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-399d09e5 Scaled up replica set payload-processing-e2e-mt-399d09e5-67bcbd677f from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-mt-399d09e5-67bcbd677f Error creating: pods "payload-processing-e2e-mt-399d09e5-67bcbd677f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-399d09e5: serviceaccount "payload-processing-e2e-mt-399d09e5" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-399d09e5-67bcbd677f Created pod: payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-399d09e5-6d9766775f Error creating: pods "payload-pre-processing-e2e-mt-399d09e5-6d9766775f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-399d09e5: serviceaccount "payload-processing-e2e-mt-399d09e5" not found 46m Normal Pulled pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Add eth0 [10.134.0.150/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-399d09e5-6d9766775f Created pod: payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg 46m Normal Started pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Container started 46m Normal Started pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Container started 46m Normal Created pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Container created 46m Normal Pulled pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Container created 46m Normal AddedInterface pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Add eth0 [10.134.0.151/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j 46m Normal AddedLabel gateway/e2e-mt-295fba7c Added label istio.io/rev=openshift-gateway to gateway e2e-mt-295fba7c 46m Normal ScalingReplicaSet deployment/e2e-mt-295fba7c-openshift-default Scaled up replica set e2e-mt-295fba7c-openshift-default-6b878475cc from 0 to 1 46m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Add eth0 [10.134.0.152/23] from ovn-kubernetes 46m Warning Unhealthy pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Readiness probe failed: Get "http://10.133.0.95:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Warning ProbeError pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Readiness probe error: Get "http://10.133.0.95:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Normal Started pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Container started 45m Warning Unhealthy pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Readiness probe failed: Get "http://10.133.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Normal Created pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Container created 45m Normal Created pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Container created 45m Normal Pulled pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Container started 45m Warning ProbeError pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Readiness probe error: Get "http://10.133.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Normal Pulled pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f Error creating: pods "payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-295fba7c: serviceaccount "payload-processing-e2e-mt-295fba7c" not found 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-295fba7c Scaled up replica set payload-pre-processing-e2e-mt-295fba7c-8d6586c7f from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-295fba7c Scaled up replica set payload-processing-e2e-mt-295fba7c-79db5944d9 from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-mt-295fba7c-79db5944d9 Error creating: pods "payload-processing-e2e-mt-295fba7c-79db5944d9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-295fba7c: serviceaccount "payload-processing-e2e-mt-295fba7c" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-295fba7c-79db5944d9 Created pod: payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f Created pod: payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt 45m Normal Started pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Container started 45m Normal AddedInterface pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Add eth0 [10.133.0.105/23] from ovn-kubernetes 45m Normal Pulled pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Add eth0 [10.134.0.153/23] from ovn-kubernetes 45m Normal Created pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Container created 45m Normal Pulled pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Container started 45m Normal Created pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Container created 45m Warning Unhealthy pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Startup probe failed: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused 45m Warning ProbeError pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Startup probe error: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Startup probe error: Get "http://10.133.0.95:15021/healthz/ready": dial tcp 10.133.0.95:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Startup probe failed: Get "http://10.133.0.95:15021/healthz/ready": dial tcp 10.133.0.95:15021: connect: connection refused 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-6d089af5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-6d089af5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-6d089af5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-6d089af5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal Killing pod/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx Stopping container payload-pre-processing 45m Normal Killing pod/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd Stopping container payload-processing 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-209c4c70-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Killing pod/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5 Stopping container payload-pre-processing 45m Normal Killing pod/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk Stopping container payload-processing 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-209c4c70-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Pulled pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Warning ProbeError pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Readiness probe error: Get "http://10.133.0.104:15021/healthz/ready": read tcp 10.133.0.2:34240->10.133.0.104:15021: read: connection reset by peer... 45m Warning Unhealthy pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Readiness probe failed: Get "http://10.133.0.104:15021/healthz/ready": read tcp 10.133.0.2:34240->10.133.0.104:15021: read: connection reset by peer 44m Normal Created pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Container created 44m Normal Started pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Container started 44m Normal Killing pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Deleted pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-dtt9j 44m Normal ScalingReplicaSet deployment/e2e-worker-w6-3b41b9-openshift-default Scaled down replica set e2e-worker-w6-3b41b9-openshift-default-84578dcc5f from 4 to 2 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f Deleted pod: e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-3b41b9-openshift-default New size: 2; reason: All metrics below target 44m Normal Killing pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-kmmmj Stopping container istio-proxy 44m Warning ProbeError pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Startup probe error: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Startup probe failed: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-399d09e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-399d09e5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q 44m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled down replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 9 to 5 44m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr Stopping container istio-proxy 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 5; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-dk5ks 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-lsjhr 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx 44m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Stopping container istio-proxy 44m Normal Killing pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr Stopping container istio-proxy 44m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-5183ae-openshift-default Scaled up replica set e2e-shared-a-w6-5183ae-openshift-default-5d948556fc from 0 to 1 44m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc Created pod: e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf 44m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-p5hrx Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal AddedLabel gateway/e2e-shared-a-w6-5183ae Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-5183ae 44m Normal AddedInterface pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Add eth0 [10.134.0.156/23] from ovn-kubernetes 44m Normal Killing pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Stopping container istio-proxy 44m Warning ProbeError pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning Unhealthy pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-399d09e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-399d09e5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-acdd08-openshift-default Scaled up replica set e2e-shared-a-w3-acdd08-openshift-default-8686dcb677 from 0 to 1 44m Normal AddedLabel gateway/e2e-shared-a-w3-acdd08 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-acdd08 44m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677 Created pod: e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g 44m Normal AddedInterface pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Add eth0 [10.133.0.109/23] from ovn-kubernetes 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9 Error creating: pods "payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae: serviceaccount "payload-processing-e2e-shared-a-w6-5183ae" not found 44m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-5bdb10-openshift-default Scaled up replica set e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c from 0 to 1 44m Normal AddedLabel gateway/e2e-shared-b-w5-5bdb10 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-5bdb10 44m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c Created pod: e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-5183ae Scaled up replica set payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc from 0 to 1 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-5183ae Scaled up replica set payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc Created pod: payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9 Created pod: payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr 44m Normal AddedLabel gateway/e2e-shared-a-w2-09fd61 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-09fd61 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc Error creating: pods "payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-5183ae: serviceaccount "payload-processing-e2e-shared-a-w6-5183ae" not found 44m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Add eth0 [10.134.0.158/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87 Created pod: e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn 44m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Container created 44m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Pulled pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Container created 44m Normal Started pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Container started 44m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Container started 44m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-09fd61-openshift-default Scaled up replica set e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87 from 0 to 1 44m Normal AddedInterface pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Add eth0 [10.134.0.157/23] from ovn-kubernetes 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Add eth0 [10.133.0.110/23] from ovn-kubernetes 44m Normal AddedInterface pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Add eth0 [10.133.0.111/23] from ovn-kubernetes 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw 44m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg 44m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr 44m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled down replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 10 to 5 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 5; reason: All metrics below target 44m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 44m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-pb2mr Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-rptcw Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled down replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 10 to 5 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 5; reason: All metrics below target 44m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p 44m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 44m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-x5tcf 44m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w Stopping container istio-proxy 44m Warning Unhealthy pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Readiness probe failed: Get "http://10.133.0.102:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z 44m Warning ProbeError pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Readiness probe error: Get "http://10.133.0.102:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled down replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 10 to 5 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 5; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-s477w 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fh9l 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-qs9jb 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c Error creating: pods "payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08: serviceaccount "payload-processing-e2e-shared-a-w3-acdd08" not found 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c Error creating: pods "payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-acdd08: serviceaccount "payload-processing-e2e-shared-a-w3-acdd08" not found 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-acdd08 Scaled up replica set payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c from 0 to 1 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-acdd08 Scaled up replica set payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c Created pod: payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 44m Normal Pulled pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Add eth0 [10.134.0.159/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c Created pod: payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd 44m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Readiness probe failed: HTTP probe failed with statuscode: 503 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Add eth0 [10.133.0.112/23] from ovn-kubernetes 44m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal Created pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Container created 44m Normal Started pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Container started 44m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Container started 44m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Container started 44m Normal Created pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Container created 44m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Container started 44m Normal Pulled pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Container created 44m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Container created 44m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Readiness probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg Readiness probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning ProbeError pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Startup probe error: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Startup probe failed: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc Created pod: payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-5bdb10 Scaled up replica set payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847 from 0 to 1 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc Error creating: pods "payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10: serviceaccount "payload-processing-e2e-shared-b-w5-5bdb10" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-5bdb10 Scaled up replica set payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc from 0 to 1 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847 Error creating: pods "payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10: serviceaccount "payload-processing-e2e-shared-b-w5-5bdb10" not found 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847 Created pod: payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf 44m Normal Created pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Container created 44m Normal Started pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Container started 44m Normal Pulled pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Startup probe failed: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused 44m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Add eth0 [10.133.0.114/23] from ovn-kubernetes 44m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Startup probe error: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused... 44m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Container started 44m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Container created 44m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Add eth0 [10.133.0.113/23] from ovn-kubernetes 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-893d83-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-893d83-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5 Created pod: payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-09fd61 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c from 0 to 1 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5 Error creating: pods "payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61: serviceaccount "payload-processing-e2e-shared-a-w2-09fd61" not found 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c Error creating: pods "payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61: serviceaccount "payload-processing-e2e-shared-a-w2-09fd61" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-09fd61 Scaled up replica set payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5 from 0 to 1 44m Normal Pulled pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Add eth0 [10.133.0.116/23] from ovn-kubernetes 44m Normal Started pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Container started 44m Normal Created pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Container created 44m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Add eth0 [10.133.0.115/23] from ovn-kubernetes 44m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Container started 44m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c Created pod: payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 44m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Container created 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-09fd61-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-pre-processing-e2e-mt-399d09e5-6d9766775f-wn6xg Stopping container payload-pre-processing 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-09fd61-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-processing-e2e-mt-399d09e5-67bcbd677f-df5vh Stopping container payload-processing 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-893d83-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-893d83-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 8; reason: All metrics below target 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled down replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 10 to 8 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 43m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled down replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 10 to 7 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 7; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qm26m Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl 43m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-8d5z7 Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lkjtc Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-64zrl Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-t75pl Stopping container istio-proxy 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 7; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled down replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 8 to 7 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-glfxl 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Readiness probe error: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Readiness probe failed: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Normal Killing pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-api-a-418f41-openshift-default Scaled up replica set e2e-api-a-418f41-openshift-default-6b76655469 from 0 to 1 43m Normal AddedLabel gateway/e2e-api-a-418f41 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-418f41 43m Normal SuccessfulCreate replicaset/e2e-api-a-418f41-openshift-default-6b76655469 Created pod: e2e-api-a-418f41-openshift-default-6b76655469-5f4st 43m Normal AddedInterface pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Add eth0 [10.133.0.122/23] from ovn-kubernetes 43m Warning ProbeError pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Readiness probe error: HTTP probe failed with statuscode: 503... 43m Warning Unhealthy pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs Readiness probe failed: HTTP probe failed with statuscode: 503 43m Warning Unhealthy pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Readiness probe failed: Get "http://10.134.0.156:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Readiness probe error: Get "http://10.134.0.156:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal Started pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Container started 43m Normal Created pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Container created 43m Normal Pulled pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-418f41 Scaled up replica set payload-processing-e2e-api-a-418f41-b6f6b688c from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-418f41-c98644447 Created pod: payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-418f41 Scaled up replica set payload-pre-processing-e2e-api-a-418f41-c98644447 from 0 to 1 42m Warning FailedCreate replicaset/payload-processing-e2e-api-a-418f41-b6f6b688c Error creating: pods "payload-processing-e2e-api-a-418f41-b6f6b688c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-418f41: serviceaccount "payload-processing-e2e-api-a-418f41" not found 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-418f41-c98644447 Error creating: pods "payload-pre-processing-e2e-api-a-418f41-c98644447-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-418f41: serviceaccount "payload-processing-e2e-api-a-418f41" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-418f41-b6f6b688c Created pod: payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd 42m Normal Started pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Container started 42m Normal AddedInterface pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Add eth0 [10.133.0.123/23] from ovn-kubernetes 42m Normal Pulled pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Container created 42m Normal Started pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Container started 42m Normal Created pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Container created 42m Normal Pulled pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Add eth0 [10.133.0.124/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Startup probe error: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Startup probe failed: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused 42m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd 42m Normal ScalingReplicaSet deployment/e2e-mt-295fba7c-openshift-default Scaled up replica set e2e-mt-295fba7c-openshift-default-6b878475cc from 1 to 2 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 42m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Container started 42m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Container created 42m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Add eth0 [10.133.0.125/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Startup probe error: Get "http://10.133.0.125:15021/healthz/ready": dial tcp 10.133.0.125:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Startup probe failed: Get "http://10.133.0.125:15021/healthz/ready": dial tcp 10.133.0.125:15021: connect: connection refused 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-5183ae-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-5183ae-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Readiness probe failed: Get "http://10.132.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 42m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Readiness probe error: Get "http://10.132.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Normal Pulled pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Container started 42m Normal Created pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Container created 42m Normal Pulled pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Container started 42m Normal Created pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Container created 42m Warning Unhealthy pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Startup probe failed: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused 42m Warning Unhealthy pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Startup probe failed: Get "http://10.133.0.109:15021/healthz/ready": dial tcp 10.133.0.109:15021: connect: connection refused 42m Warning ProbeError pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Startup probe error: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused... 42m Warning ProbeError pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Startup probe error: Get "http://10.133.0.109:15021/healthz/ready": dial tcp 10.133.0.109:15021: connect: connection refused... 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-5bdb10-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-5bdb10-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Normal AddedLabel gateway/e2e-shared-b-w6-32b500 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-32b500 42m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-32b500-openshift-default-675579b8b9 Created pod: e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks 42m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-32b500-openshift-default Scaled up replica set e2e-shared-b-w6-32b500-openshift-default-675579b8b9 from 0 to 1 42m Normal AddedInterface pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Add eth0 [10.132.0.96/23] from ovn-kubernetes 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-acdd08-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-acdd08-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-32b500 Scaled up replica set payload-processing-e2e-shared-b-w6-32b500-6f68bbff87 from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-32b500 Scaled up replica set payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc Error creating: pods "payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-32b500: serviceaccount "payload-processing-e2e-shared-b-w6-32b500" not found 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87 Created pod: payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz 41m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77 Created pod: e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr 41m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87 Error creating: pods "payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-32b500: serviceaccount "payload-processing-e2e-shared-b-w6-32b500" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc Created pod: payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv 41m Normal AddedLabel gateway/e2e-shared-b-w3-1fa30f Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-1fa30f 41m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-1fa30f-openshift-default Scaled up replica set e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77 from 0 to 1 41m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Container started 41m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Container created 41m Normal Started pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Container started 41m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Add eth0 [10.134.0.171/23] from ovn-kubernetes 41m Normal AddedInterface pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Add eth0 [10.132.0.97/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Container created 41m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Add eth0 [10.133.0.126/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf 41m Normal AddedLabel gateway/e2e-inf-a-72a642 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-72a642 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 4; reason: All metrics below target 41m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled down replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 5 to 4 41m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-m82xf Stopping container istio-proxy 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-32b500-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-32b500-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal ScalingReplicaSet deployment/e2e-inf-a-72a642-openshift-default Scaled up replica set e2e-inf-a-72a642-openshift-default-6f59f57b79 from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-a-72a642-openshift-default-6f59f57b79 Created pod: e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 41m Normal AddedInterface pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Add eth0 [10.134.0.172/23] from ovn-kubernetes 41m Normal AddedLabel gateway/e2e-shared-b-w2-8e6dc7 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-8e6dc7 41m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99 Created pod: e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 41m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-8e6dc7-openshift-default Scaled up replica set e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99 from 0 to 1 41m Normal AddedInterface pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Add eth0 [10.132.0.98/23] from ovn-kubernetes 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-acdd08-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-acdd08-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47 Created pod: payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47 Error creating: pods "payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f: serviceaccount "payload-processing-e2e-shared-b-w3-1fa30f" not found 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-1fa30f Scaled up replica set payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47 from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-1fa30f Scaled up replica set payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff from 0 to 1 41m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff Error creating: pods "payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-1fa30f: serviceaccount "payload-processing-e2e-shared-b-w3-1fa30f" not found 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff Created pod: payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk 41m Normal Pulled pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Container created 41m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Add eth0 [10.134.0.175/23] from ovn-kubernetes 41m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Container started 41m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Add eth0 [10.134.0.174/23] from ovn-kubernetes 41m Normal Created pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Container created 41m Normal Started pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Container started 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-5bdb10-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-5bdb10-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-72a642-6d8f478c7 Created pod: payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-72a642-6d8f478c7 Error creating: pods "payload-processing-e2e-inf-a-72a642-6d8f478c7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-72a642: serviceaccount "payload-processing-e2e-inf-a-72a642" not found 41m Normal Started pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Container started 41m Normal Created pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Container created 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-72a642 Scaled up replica set payload-processing-e2e-inf-a-72a642-6d8f478c7 from 0 to 1 41m Normal Pulled pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f Error creating: pods "payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-72a642: serviceaccount "payload-processing-e2e-inf-a-72a642" not found 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-72a642 Scaled up replica set payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f Created pod: payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Add eth0 [10.134.0.176/23] from ovn-kubernetes 41m Normal Created pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Container created 41m Normal Started pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Container started 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Add eth0 [10.134.0.177/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Container started 41m Normal Created pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Container created 41m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc Error creating: pods "payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7: serviceaccount "payload-processing-e2e-shared-b-w2-8e6dc7" not found 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-8e6dc7 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47 from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-8e6dc7 Scaled up replica set payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47 Created pod: payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47 Error creating: pods "payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7: serviceaccount "payload-processing-e2e-shared-b-w2-8e6dc7" not found 41m Warning ProbeError pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Startup probe error: Get "http://10.133.0.111:15021/healthz/ready": dial tcp 10.133.0.111:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Startup probe failed: Get "http://10.133.0.111:15021/healthz/ready": dial tcp 10.133.0.111:15021: connect: connection refused 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc Created pod: payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 41m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Add eth0 [10.133.0.128/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Container started 41m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Container started 41m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Add eth0 [10.133.0.129/23] from ovn-kubernetes 41m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Container created 41m Normal Pulled pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Container created 41m Normal Pulled pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Container started 41m Normal Created pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Container created 41m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 4; reason: All metrics below target 41m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-46c79 Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled down replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 5 to 4 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-418f41-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-418f41-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning ProbeError pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Startup probe error: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Startup probe failed: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-09fd61-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-09fd61-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-418f41-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-418f41-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal Killing pod/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw Stopping container payload-processing 40m Normal Killing pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-82stt Stopping container payload-pre-processing 40m Normal Pulled pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Started pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Container started 40m Normal Created pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Container created 40m Normal AddedLabel gateway/e2e-api-a-f6d9a2 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-f6d9a2 40m Normal SuccessfulCreate replicaset/e2e-api-a-f6d9a2-openshift-default-5877dc88df Created pod: e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k 40m Normal ScalingReplicaSet deployment/e2e-api-a-f6d9a2-openshift-default Scaled up replica set e2e-api-a-f6d9a2-openshift-default-5877dc88df from 0 to 1 40m Normal AddedInterface pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Add eth0 [10.134.0.179/23] from ovn-kubernetes 40m Warning ProbeError pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Startup probe error: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Startup probe failed: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-f6d9a2 Scaled up replica set payload-pre-processing-e2e-api-a-f6d9a2-7847977f47 from 0 to 1 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-f6d9a2 Scaled up replica set payload-processing-e2e-api-a-f6d9a2-6d77b94775 from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47 Created pod: payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47 Error creating: pods "payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f6d9a2: serviceaccount "payload-processing-e2e-api-a-f6d9a2" not found 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-f6d9a2-6d77b94775 Created pod: payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k 39m Warning FailedCreate replicaset/payload-processing-e2e-api-a-f6d9a2-6d77b94775 Error creating: pods "payload-processing-e2e-api-a-f6d9a2-6d77b94775-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f6d9a2: serviceaccount "payload-processing-e2e-api-a-f6d9a2" not found 39m Normal AddedInterface pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Add eth0 [10.134.0.182/23] from ovn-kubernetes 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-32b500-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Pulled pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-32b500-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Created pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Container created 39m Normal Started pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Container started 39m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Add eth0 [10.134.0.181/23] from ovn-kubernetes 39m Normal Pulled pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Created pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Container created 39m Normal Started pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Container started 39m Normal Killing pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Stopping container payload-pre-processing 39m Normal Killing pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Stopping container payload-processing 39m Warning Unhealthy pod/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd Readiness probe failed: timeout: failed to connect service "10.133.0.123:50052" within 1s: context deadline exceeded 39m Warning Unhealthy pod/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g Readiness probe failed: timeout: failed to connect service "10.133.0.124:50052" within 1s: context deadline exceeded 39m Normal Started pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Container started 39m Normal Created pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Container created 39m Normal Pulled pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Pulled pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Container created 39m Normal Started pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Container started 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-1fa30f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-1fa30f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Stopping container istio-proxy 39m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled down replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 5 to 4 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 3; reason: All metrics below target 39m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled down replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 5 to 3 39m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb 39m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 4; reason: All metrics below target 39m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg 39m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Stopping container istio-proxy 39m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Stopping container istio-proxy 39m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s78xr Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-bmwwg Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 39m Warning ProbeError pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Startup probe error: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Startup probe failed: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning Unhealthy pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-qjxsb Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-72a642-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-72a642-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning Unhealthy pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Readiness probe failed: Get "http://10.132.0.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Warning ProbeError pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Readiness probe error: Get "http://10.132.0.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Normal Created pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Container created 38m Normal Pulled pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Started pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Container started 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-1fa30f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-1fa30f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning ProbeError pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Startup probe error: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused... 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc 38m Warning Unhealthy pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Startup probe failed: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused 38m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 38m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled down replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 7 to 4 38m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Stopping container istio-proxy 38m Normal ScalingReplicaSet deployment/e2e-worker-w5-3fea3a-openshift-default Scaled down replica set e2e-worker-w5-3fea3a-openshift-default-7df4bff97 from 4 to 3 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3fea3a-openshift-default New size: 3; reason: All metrics below target 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 4; reason: All metrics below target 38m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx 38m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc Stopping container istio-proxy 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 3; reason: All metrics below target 38m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-vlnf6 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 4; reason: All metrics below target 38m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled down replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 4 to 3 38m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw 38m Normal SuccessfulDelete replicaset/e2e-worker-w5-3fea3a-openshift-default-7df4bff97 Deleted pod: e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz 38m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Stopping container istio-proxy 38m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled down replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 7 to 4 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m 38m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg Stopping container istio-proxy 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg 38m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Readiness probe error: HTTP probe failed with statuscode: 503... 38m Normal Killing pod/payload-processing-e2e-shared-a-w6-5183ae-6b55b7cdf9-r6mpr Stopping container payload-processing 38m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Readiness probe failed: HTTP probe failed with statuscode: 503 38m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl Stopping container payload-pre-processing 38m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-hbhl8 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-fxdnx Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning Unhealthy pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-8e6dc7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-8e6dc7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9 Stopping container payload-pre-processing 38m Normal Killing pod/payload-processing-e2e-shared-a-w3-acdd08-754dbfc48c-w2nfd Stopping container payload-processing 38m Normal Killing pod/payload-processing-e2e-inf-a-72a642-6d8f478c7-xfzbm Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-inf-a-72a642-7fc9586d5f-rtjrs Stopping container payload-pre-processing 38m Normal Killing pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25 Stopping container payload-pre-processing 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-8e6dc7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-8e6dc7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning Unhealthy pod/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5 Readiness probe failed: timeout: failed to connect service "10.133.0.115:50052" within 1s: context deadline exceeded 38m Normal Killing pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st Stopping container istio-proxy 38m Normal SuccessfulCreate replicaset/e2e-inf-a-0e8bf4-openshift-default-575655668f Created pod: e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s 38m Normal AddedLabel gateway/e2e-inf-a-0e8bf4 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-0e8bf4 38m Normal ScalingReplicaSet deployment/e2e-inf-a-0e8bf4-openshift-default Scaled up replica set e2e-inf-a-0e8bf4-openshift-default-575655668f from 0 to 1 38m Normal AddedInterface pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Add eth0 [10.133.0.133/23] from ovn-kubernetes 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-0e8bf4 Scaled up replica set payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-0e8bf4 Scaled up replica set payload-processing-e2e-inf-a-0e8bf4-7f5db668cf from 0 to 1 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf Error creating: pods "payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0e8bf4: serviceaccount "payload-processing-e2e-inf-a-0e8bf4" not found 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc Created pod: payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Add eth0 [10.133.0.134/23] from ovn-kubernetes 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc Error creating: pods "payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0e8bf4: serviceaccount "payload-processing-e2e-inf-a-0e8bf4" not found 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf Created pod: payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 38m Normal Started pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Container started 38m Normal Created pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Add eth0 [10.133.0.135/23] from ovn-kubernetes 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Pulled pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Container created 38m Normal Started pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Container started 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 3; reason: All metrics below target 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt 37m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Stopping container istio-proxy 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-0e8bf4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-0e8bf4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default Scaled down replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 4 to 3 37m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-295fba7c Scaled up replica set payload-pre-processing-e2e-mt-295fba7c-8d6586c7f from 0 to 1 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f Error creating: pods "payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-295fba7c: serviceaccount "payload-processing-e2e-mt-295fba7c" not found 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-295fba7c Scaled up replica set payload-processing-e2e-mt-295fba7c-79db5944d9 from 0 to 1 37m Warning FailedCreate replicaset/payload-processing-e2e-mt-295fba7c-79db5944d9 Error creating: pods "payload-processing-e2e-mt-295fba7c-79db5944d9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-295fba7c: serviceaccount "payload-processing-e2e-mt-295fba7c" not found 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-295fba7c-79db5944d9 Created pod: payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f Created pod: payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v 37m Normal AddedInterface pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Add eth0 [10.133.0.137/23] from ovn-kubernetes 37m Normal AddedInterface pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Add eth0 [10.133.0.136/23] from ovn-kubernetes 37m Normal Created pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Container created 37m Normal Pulled pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal Started pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Container started 37m Normal Started pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Container started 37m Normal Pulled pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal Created pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Container created 37m Normal Killing pod/e2e-shared-a-w6-5183ae-openshift-default-5d948556fc-2ghmf Stopping container istio-proxy 37m Normal Killing pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g Stopping container istio-proxy 37m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv Stopping container payload-pre-processing 37m Normal Killing pod/payload-processing-e2e-shared-b-w6-32b500-6f68bbff87-qlcvz Stopping container payload-processing 37m Normal Killing pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Readiness probe failed: Get "http://10.134.0.179:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Readiness probe error: Get "http://10.134.0.179:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Normal Created pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Container created 37m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-1fa30f-5d5d4c7d47-8m78b Stopping container payload-pre-processing 37m Normal Pulled pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Killing pod/payload-processing-e2e-shared-b-w3-1fa30f-76c57885ff-tnlpk Stopping container payload-processing 37m Normal Started pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Container started 37m Warning Unhealthy pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Startup probe failed: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused 37m Warning ProbeError pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Startup probe error: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused... 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f6d9a2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk Stopping container payload-pre-processing 37m Normal Killing pod/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8 Stopping container payload-processing 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f6d9a2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal Killing pod/e2e-inf-a-72a642-openshift-default-6f59f57b79-7mzb2 Stopping container istio-proxy 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f6d9a2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f6d9a2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal Killing pod/payload-processing-e2e-shared-a-w5-893d83-b64d457d5-vf8jv Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-893d83-799fb896f5-pv8sl Stopping container payload-pre-processing 36m Normal Killing pod/payload-processing-e2e-api-a-f6d9a2-6d77b94775-t6m5k Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-api-a-f6d9a2-7847977f47-mh9mt Stopping container payload-pre-processing 36m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Stopping container istio-proxy 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 2; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 36m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default New size: 2; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr 36m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-jkw59 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Stopping container istio-proxy 36m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-88kwr Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 36m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 36m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-wrngs Stopping container discovery 36m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-wrngs 36m Normal Killing pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Stopping container istio-proxy 36m Warning Unhealthy pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-shared-b-w6-32b500-openshift-default-675579b8b9-8p8ks Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal Killing pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Stopping container payload-processing 35m Normal Killing pod/e2e-shared-b-w3-1fa30f-openshift-default-56549fbd77-j7zjr Stopping container istio-proxy 35m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf Stopping container payload-pre-processing 35m Normal AddedLabel gateway/e2e-api-a-a8983d Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-a8983d 35m Normal ScalingReplicaSet deployment/e2e-api-a-a8983d-openshift-default Scaled up replica set e2e-api-a-a8983d-openshift-default-57b8f8f4c9 from 0 to 1 35m Normal SuccessfulCreate replicaset/e2e-api-a-a8983d-openshift-default-57b8f8f4c9 Created pod: e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc 35m Normal AddedInterface pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Add eth0 [10.133.0.140/23] from ovn-kubernetes 35m Normal Killing pod/e2e-shared-b-w2-8e6dc7-openshift-default-cffdc5c99-86tm8 Stopping container istio-proxy 35m Warning Unhealthy pod/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2 Readiness probe failed: timeout: failed to connect service "10.133.0.114:50052" within 1s: context deadline exceeded 35m Normal AddedLabel gateway/e2e-api-a-1fb4bc Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-1fb4bc 35m Normal SuccessfulCreate replicaset/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49 Created pod: e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb 35m Normal ScalingReplicaSet deployment/e2e-api-a-1fb4bc-openshift-default Scaled up replica set e2e-api-a-1fb4bc-openshift-default-5c9fc7df49 from 0 to 1 35m Normal Killing pod/e2e-api-a-f6d9a2-openshift-default-5877dc88df-mgv6k Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-a8983d Scaled up replica set payload-processing-e2e-api-a-a8983d-855dc8669c from 0 to 1 35m Warning FailedCreate replicaset/payload-processing-e2e-api-a-a8983d-855dc8669c Error creating: pods "payload-processing-e2e-api-a-a8983d-855dc8669c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a8983d: serviceaccount "payload-processing-e2e-api-a-a8983d" not found 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-a8983d-855dc8669c Created pod: payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-a8983d Scaled up replica set payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9 from 0 to 1 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9 Error creating: pods "payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a8983d: serviceaccount "payload-processing-e2e-api-a-a8983d" not found 35m Normal AddedInterface pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Add eth0 [10.133.0.141/23] from ovn-kubernetes 35m Normal AddedLabel gateway/e2e-inf-a-50ae96 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-50ae96 35m Normal AddedInterface pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Add eth0 [10.133.0.142/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f Created pod: e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9 Created pod: payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv 35m Normal ScalingReplicaSet deployment/e2e-inf-a-50ae96-openshift-default Scaled up replica set e2e-inf-a-50ae96-openshift-default-65c7ddfc5f from 0 to 1 35m Normal Created pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Container created 35m Normal Created pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Container created 35m Normal Started pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Container started 35m Normal Pulled pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Add eth0 [10.133.0.144/23] from ovn-kubernetes 35m Normal Started pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Container started 35m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Add eth0 [10.133.0.143/23] from ovn-kubernetes 35m Normal Pulled pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-zvjvn 35m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-zvjvn Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-zlfl6 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-fvch8 35m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: All metrics below target 35m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-zlfl6 Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 10 to 6 35m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-tmkwr 35m Normal AddedLabel gateway/e2e-inf-b-dcdb23 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-dcdb23 35m Normal AddedInterface pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Add eth0 [10.132.0.102/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-inf-b-dcdb23-openshift-default-6df5cd9845 Created pod: e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 35m Normal ScalingReplicaSet deployment/e2e-inf-b-dcdb23-openshift-default Scaled up replica set e2e-inf-b-dcdb23-openshift-default-6df5cd9845 from 0 to 1 35m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-tmkwr Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-1fb4bc Scaled up replica set payload-processing-e2e-api-a-1fb4bc-5fff7d7475 from 0 to 1 35m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Readiness probe error: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Normal Started pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Container started 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-1fb4bc-5fff7d7475 Created pod: payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c Error creating: pods "payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-1fb4bc: serviceaccount "payload-processing-e2e-api-a-1fb4bc" not found 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c Created pod: payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 35m Normal Created pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Container created 35m Warning FailedCreate replicaset/payload-processing-e2e-api-a-1fb4bc-5fff7d7475 Error creating: pods "payload-processing-e2e-api-a-1fb4bc-5fff7d7475-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-1fb4bc: serviceaccount "payload-processing-e2e-api-a-1fb4bc" not found 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-1fb4bc Scaled up replica set payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c from 0 to 1 35m Normal Pulled pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-fvch8 Readiness probe failed: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Add eth0 [10.132.0.103/23] from ovn-kubernetes 35m Normal AddedInterface pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Add eth0 [10.132.0.104/23] from ovn-kubernetes 35m Normal Pulling pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 35m Normal Pulling pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-50ae96 Scaled up replica set payload-processing-e2e-inf-a-50ae96-667dc45bb9 from 0 to 1 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-50ae96 Scaled up replica set payload-pre-processing-e2e-inf-a-50ae96-fcf4b47 from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47 Created pod: payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47 Error creating: pods "payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-50ae96: serviceaccount "payload-processing-e2e-inf-a-50ae96" not found 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-50ae96-667dc45bb9 Created pod: payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g 35m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-50ae96-667dc45bb9 Error creating: pods "payload-processing-e2e-inf-a-50ae96-667dc45bb9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-50ae96: serviceaccount "payload-processing-e2e-inf-a-50ae96" not found 35m Normal Pulled pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Add eth0 [10.132.0.105/23] from ovn-kubernetes 35m Normal Pulling pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 35m Normal Created pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Container created 35m Normal Started pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Container started 35m Normal Pulled pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.519s (3.519s including waiting). Image size: 125353963 bytes. 35m Normal Created pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Container created 35m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Add eth0 [10.133.0.145/23] from ovn-kubernetes 35m Normal Pulled pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.408s (3.408s including waiting). Image size: 125353963 bytes. 35m Normal Created pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Container created 35m Normal Started pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Container started 35m Normal Pulled pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 815ms (816ms including waiting). Image size: 125353963 bytes. 35m Normal Started pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Container started 35m Warning ProbeError pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Startup probe error: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Startup probe failed: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused 35m Normal Started pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Container started 35m Normal Created pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Container created 35m Normal ScalingReplicaSet deployment/e2e-mt-295fba7c-openshift-default Scaled up replica set e2e-mt-295fba7c-openshift-default-6b878475cc from 2 to 4 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Add eth0 [10.132.0.107/23] from ovn-kubernetes 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Container started 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Container created 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Container started 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Add eth0 [10.132.0.106/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Container created 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-dcdb23-56c7f9c647 Created pod: payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-dcdb23 Scaled up replica set payload-processing-e2e-inf-b-dcdb23-56c7f9c647 from 0 to 1 35m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-dcdb23-56c7f9c647 Error creating: pods "payload-processing-e2e-inf-b-dcdb23-56c7f9c647-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-dcdb23: serviceaccount "payload-processing-e2e-inf-b-dcdb23" not found 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-dcdb23 Scaled up replica set payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7 from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7 Created pod: payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7 Error creating: pods "payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-dcdb23: serviceaccount "payload-processing-e2e-inf-b-dcdb23" not found 35m Normal Started pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Container started 35m Normal Created pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Container created 35m Normal Pulled pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Add eth0 [10.134.0.194/23] from ovn-kubernetes 35m Normal Pulled pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Created pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Container created 35m Normal Started pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Container started 35m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Add eth0 [10.133.0.146/23] from ovn-kubernetes 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Startup probe error: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Startup probe failed: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Startup probe error: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Startup probe failed: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Add eth0 [10.133.0.147/23] from ovn-kubernetes 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Add eth0 [10.134.0.196/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/e2e-mt-295fba7c-openshift-default Scaled up replica set e2e-mt-295fba7c-openshift-default-6b878475cc from 4 to 8 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default New size: 8; reason: 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Container started 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Container created 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Container started 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Container created 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Add eth0 [10.132.0.109/23] from ovn-kubernetes 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Container started 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Add eth0 [10.133.0.148/23] from ovn-kubernetes 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Container started 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Container created 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Container created 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Startup probe error: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Startup probe failed: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Startup probe error: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Startup probe failed: Get "http://10.133.0.148:15021/healthz/ready": dial tcp 10.133.0.148:15021: connect: connection refused 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Startup probe error: Get "http://10.133.0.148:15021/healthz/ready": dial tcp 10.133.0.148:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Startup probe failed: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused 35m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Startup probe error: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Startup probe failed: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused 35m Normal Killing pod/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v Stopping container payload-pre-processing 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-0e8bf4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-0e8bf4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Add eth0 [10.134.0.197/23] from ovn-kubernetes 35m Normal AddedInterface pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Add eth0 [10.132.0.110/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 35m Normal SuccessfulCreate replicaset/e2e-mt-295fba7c-openshift-default-6b878475cc (combined from similar events): Created pod: e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-295fba7c-openshift-default New size: 10; reason: 35m Normal ScalingReplicaSet deployment/e2e-mt-295fba7c-openshift-default Scaled up replica set e2e-mt-295fba7c-openshift-default-6b878475cc from 8 to 10 35m Normal Killing pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Stopping container istio-proxy 35m Warning ProbeError pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-shared-b-w5-5bdb10-openshift-default-7d84fd996c-l6hr9 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Container created 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Container started 35m Normal Created pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Container created 35m Normal Pulled pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Started pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Container started 34m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Startup probe failed: Get "http://10.132.0.110:15021/healthz/ready": dial tcp 10.132.0.110:15021: connect: connection refused 34m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Startup probe failed: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused 34m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Startup probe error: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused... 34m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Startup probe error: Get "http://10.132.0.110:15021/healthz/ready": dial tcp 10.132.0.110:15021: connect: connection refused... 34m Normal Killing pod/payload-processing-e2e-worker-w5-3fea3a-5f7fd56f87-t7x7l Stopping container payload-processing 34m Normal Killing pod/payload-pre-processing-e2e-worker-w5-3fea3a-5474b45b7-qvw2r Stopping container payload-pre-processing 34m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default Scaled down replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 4 to 3 34m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb 34m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-gfxpb Stopping container istio-proxy 34m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled down replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 4 to 3 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 3; reason: All metrics below target 34m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc 34m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc Stopping container istio-proxy 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 3; reason: All metrics below target 34m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Stopping container istio-proxy 34m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz 34m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default Scaled down replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 3 to 2 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 2; reason: All metrics below target 34m Warning Unhealthy pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Readiness probe failed: HTTP probe failed with statuscode: 503 34m Warning ProbeError pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-zvzgz Readiness probe error: HTTP probe failed with statuscode: 503... 34m Normal Started pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Container started 34m Warning Unhealthy pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Readiness probe failed: Get "http://10.133.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Readiness probe error: Get "http://10.133.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal Pulled pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Container created 34m Normal Pulled pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Container started 34m Normal Created pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Container created 34m Normal Pulled pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Container started 34m Normal Created pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Container created 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-a8983d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-a8983d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning Unhealthy pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Startup probe failed: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused 33m Warning ProbeError pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Startup probe error: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Startup probe failed: Get "http://10.133.0.140:15021/healthz/ready": dial tcp 10.133.0.140:15021: connect: connection refused 33m Warning ProbeError pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Startup probe error: Get "http://10.133.0.140:15021/healthz/ready": dial tcp 10.133.0.140:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Startup probe failed: Get "http://10.133.0.141:15021/healthz/ready": dial tcp 10.133.0.141:15021: connect: connection refused 33m Warning ProbeError pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Startup probe error: Get "http://10.133.0.141:15021/healthz/ready": dial tcp 10.133.0.141:15021: connect: connection refused... 33m Normal Killing pod/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp Stopping container payload-processing 33m Normal Killing pod/payload-pre-processing-e2e-worker-w6-3b41b9-7d8f58b86c-gj8x6 Stopping container payload-pre-processing 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-1fb4bc-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-1fb4bc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-50ae96-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-50ae96-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal SuccessfulCreate replicaset/e2e-api-b-18b89b-openshift-default-7bdb7588f9 Created pod: e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp 33m Normal AddedLabel gateway/e2e-api-b-18b89b Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-18b89b 33m Normal ScalingReplicaSet deployment/e2e-api-b-18b89b-openshift-default Scaled up replica set e2e-api-b-18b89b-openshift-default-7bdb7588f9 from 0 to 1 33m Normal AddedInterface pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Add eth0 [10.134.0.202/23] from ovn-kubernetes 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-1fb4bc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-1fb4bc-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-50ae96-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-50ae96-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal AddedLabel gateway/e2e-api-b-44c08f Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-44c08f 33m Normal SuccessfulCreate replicaset/e2e-api-b-44c08f-openshift-default-86db5cb557 Created pod: e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm 33m Normal ScalingReplicaSet deployment/e2e-api-b-44c08f-openshift-default Scaled up replica set e2e-api-b-44c08f-openshift-default-86db5cb557 from 0 to 1 33m Normal AddedInterface pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Add eth0 [10.134.0.203/23] from ovn-kubernetes 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7 Error creating: pods "payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-18b89b: serviceaccount "payload-processing-e2e-api-b-18b89b" not found 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7 Created pod: payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s 33m Warning FailedCreate replicaset/payload-processing-e2e-api-b-18b89b-74bcfbd577 Error creating: pods "payload-processing-e2e-api-b-18b89b-74bcfbd577-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-18b89b: serviceaccount "payload-processing-e2e-api-b-18b89b" not found 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-18b89b Scaled up replica set payload-processing-e2e-api-b-18b89b-74bcfbd577 from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-18b89b Scaled up replica set payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7 from 0 to 1 33m Normal Started pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Container started 33m Normal AddedInterface pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Add eth0 [10.133.0.152/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-18b89b-74bcfbd577 Created pod: payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm 33m Normal Created pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Container created 33m Normal Started pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Container started 33m Normal Created pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Container created 33m Normal Pulled pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Add eth0 [10.133.0.151/23] from ovn-kubernetes 33m Normal Pulled pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Warning ProbeError pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Readiness probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Readiness probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-a8983d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Created pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Container created 33m Normal Pulled pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Container started 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-a8983d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal AddedLabel gateway/e2e-inf-b-eac348 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-eac348 33m Normal AddedInterface pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Add eth0 [10.133.0.153/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/e2e-inf-b-eac348-openshift-default-7ff86ddc97 Created pod: e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z 33m Normal ScalingReplicaSet deployment/e2e-inf-b-eac348-openshift-default Scaled up replica set e2e-inf-b-eac348-openshift-default-7ff86ddc97 from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-44c08f Scaled up replica set payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c from 0 to 1 33m Warning FailedCreate replicaset/payload-processing-e2e-api-b-44c08f-5c7554d75f Error creating: pods "payload-processing-e2e-api-b-44c08f-5c7554d75f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-44c08f: serviceaccount "payload-processing-e2e-api-b-44c08f" not found 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-44c08f Scaled up replica set payload-processing-e2e-api-b-44c08f-5c7554d75f from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c Created pod: payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn 33m Warning ProbeError pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Startup probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Startup probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 33m Normal Started pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Container started 33m Normal Created pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Container created 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c Error creating: pods "payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-44c08f: serviceaccount "payload-processing-e2e-api-b-44c08f" not found 33m Normal Pulled pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Add eth0 [10.134.0.204/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-44c08f-5c7554d75f Created pod: payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h 33m Normal Started pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Container started 33m Normal AddedInterface pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Add eth0 [10.134.0.205/23] from ovn-kubernetes 33m Normal Pulled pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Container created 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-eac348 Scaled up replica set payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75 from 0 to 1 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75 Error creating: pods "payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-eac348: serviceaccount "payload-processing-e2e-inf-b-eac348" not found 33m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-eac348-cfd67fd55 Error creating: pods "payload-processing-e2e-inf-b-eac348-cfd67fd55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-eac348: serviceaccount "payload-processing-e2e-inf-b-eac348" not found 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-eac348-cfd67fd55 Created pod: payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-eac348 Scaled up replica set payload-processing-e2e-inf-b-eac348-cfd67fd55 from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75 Created pod: payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-dcdb23-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-dcdb23-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Created pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Container created 33m Normal Started pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Container started 33m Normal Pulled pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Add eth0 [10.134.0.207/23] from ovn-kubernetes 33m Normal Started pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Container started 33m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Add eth0 [10.134.0.206/23] from ovn-kubernetes 33m Normal Pulled pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Container created 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-18b89b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-18b89b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jp2cv Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-wcjth Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-rbmdl Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4lmml Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-4sjw9 Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-9nkfj Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd Stopping container istio-proxy 33m Normal Killing pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw Stopping container istio-proxy 33m Warning Unhealthy pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-mt-295fba7c-openshift-default-6b878475cc-f4w2j Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-eac348-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-eac348-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s Stopping container istio-proxy 32m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-9sltf Stopping container istio-proxy 32m Normal Killing pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Stopping container istio-proxy 32m Warning ProbeError pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-m6p2g Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-dcdb23-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-dcdb23-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 6 to 10 32m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 32m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Container started 32m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Container created 32m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Container created 32m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Add eth0 [10.134.0.210/23] from ovn-kubernetes 32m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Add eth0 [10.132.0.113/23] from ovn-kubernetes 32m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Add eth0 [10.132.0.114/23] from ovn-kubernetes 32m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Add eth0 [10.132.0.115/23] from ovn-kubernetes 32m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Container started 32m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-5pm4h 32m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Container created 32m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-8xt57 32m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-gbst7 32m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-qbnpn 32m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Container started 32m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Container created 32m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Container started 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Startup probe failed: Get "http://10.134.0.210:15021/healthz/ready": dial tcp 10.134.0.210:15021: connect: connection refused 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Startup probe failed: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Startup probe error: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused... 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Startup probe error: Get "http://10.134.0.210:15021/healthz/ready": dial tcp 10.134.0.210:15021: connect: connection refused... 32m Warning ProbeError pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Readiness probe error: Get "http://10.134.0.202:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Startup probe failed: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Startup probe failed: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Startup probe error: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Readiness probe failed: Get "http://10.134.0.202:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Startup probe error: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused... 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Startup probe error: Get "http://10.132.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 32m Normal Pulled pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Container started 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Startup probe failed: Get "http://10.132.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Normal Created pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Container created 31m Warning ProbeError pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Startup probe error: Get "http://10.134.0.202:15021/healthz/ready": dial tcp 10.134.0.202:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Startup probe failed: Get "http://10.134.0.202:15021/healthz/ready": dial tcp 10.134.0.202:15021: connect: connection refused 31m Normal Killing pod/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59 Stopping container payload-pre-processing 31m Warning Unhealthy pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Readiness probe failed: Get "http://10.134.0.203:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 31m Warning ProbeError pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Readiness probe error: Get "http://10.134.0.203:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 31m Normal Killing pod/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw Stopping container payload-processing 31m Normal Started pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Container started 31m Normal Started pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Container started 31m Normal Pulled pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Container created 31m Normal Created pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Container created 31m Normal Pulled pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-18b89b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-18b89b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning Unhealthy pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Startup probe failed: Get "http://10.134.0.203:15021/healthz/ready": dial tcp 10.134.0.203:15021: connect: connection refused 31m Warning ProbeError pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Startup probe error: Get "http://10.134.0.203:15021/healthz/ready": dial tcp 10.134.0.203:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Startup probe failed: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused 31m Warning ProbeError pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Startup probe error: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused... 31m Normal Killing pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Stopping container istio-proxy 31m Normal Killing pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 Stopping container istio-proxy 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-44c08f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-44c08f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning Unhealthy pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-f6zxg Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-eac348-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-eac348-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-44c08f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal Killing pod/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv Stopping container payload-pre-processing 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-44c08f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal Killing pod/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74 Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-api-a-1fb4bc-5b8d56554c-65n54 Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-api-a-1fb4bc-5fff7d7475-zz2q7 Stopping container payload-processing 30m Normal ScalingReplicaSet deployment/e2e-worker-w4-ffee69-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w4-ffee69-openshift-default-86bcc884c from 2 to 1 30m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Stopping container istio-proxy 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-ffee69-openshift-default (combined from similar events): New size: 1; reason: All metrics below target 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default (combined from similar events): New size: 1; reason: All metrics below target 30m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g Stopping container istio-proxy 30m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 2 to 1 30m Normal SuccessfulDelete replicaset/e2e-worker-w1-9697f7-openshift-default-74d989748c Deleted pod: e2e-worker-w1-9697f7-openshift-default-74d989748c-mlh8g 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9697f7-openshift-default New size: 2; reason: All metrics below target 30m Normal SuccessfulDelete replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Deleted pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k 30m Normal ScalingReplicaSet deployment/e2e-worker-w1-9697f7-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-9697f7-openshift-default-74d989748c from 3 to 2 30m Normal SuccessfulDelete replicaset/e2e-worker-w4-ffee69-openshift-default-86bcc884c Deleted pod: e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 30m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-99p9k Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning Unhealthy pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-2fs55 Readiness probe error: HTTP probe failed with statuscode: 503... 30m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 Stopping container istio-proxy 30m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default Scaled down replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 3 to 2 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 2; reason: All metrics below target 30m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-qjrx7 30m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lhzv6 Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-z92fh Stopping container istio-proxy 29m Normal Killing pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc Stopping container istio-proxy 29m Normal Killing pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb Stopping container istio-proxy 29m Normal Killing pod/payload-processing-e2e-api-b-44c08f-5c7554d75f-4g42h Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-api-b-44c08f-5f8c54bf8c-w86cn Stopping container payload-pre-processing 29m Normal Killing pod/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm Stopping container payload-processing 28m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr Stopping container istio-proxy 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6606d6-openshift-default New size: 1; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-worker-w0-6606d6-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w0-6606d6-openshift-default-86bfc64849 from 2 to 1 28m Normal SuccessfulDelete replicaset/e2e-worker-w2-669d6b-openshift-default-548f97d447 Deleted pod: e2e-worker-w2-669d6b-openshift-default-548f97d447-m2nsr 28m Normal ScalingReplicaSet deployment/e2e-worker-w2-669d6b-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-669d6b-openshift-default-548f97d447 from 2 to 1 28m Normal SuccessfulDelete replicaset/e2e-worker-w0-6606d6-openshift-default-86bfc64849 Deleted pod: e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv 28m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv Stopping container istio-proxy 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-669d6b-openshift-default New size: 1; reason: All metrics below target 27m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-47mcc 27m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 27m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-47mcc Stopping container discovery 27m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 27m Normal Killing pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Stopping container istio-proxy 27m Warning Unhealthy pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-api-b-18b89b-openshift-default-7bdb7588f9-jk8fp Readiness probe error: HTTP probe failed with statuscode: 503... 27m Normal Killing pod/e2e-api-b-44c08f-openshift-default-86db5cb557-mjxhm Stopping container istio-proxy 27m Normal Killing pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-worker-w2-669d6b-58ddc8f7bc-v2fnk Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w2-669d6b-86d46969f7-pkr6v Stopping container payload-processing 27m Warning Unhealthy pod/payload-processing-e2e-worker-w3-1f2e7e-6cb6659785-lq5x2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-1f2e7e-7d4d8fbd9f-8nxrq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Normal Killing pod/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9 Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Stopping container payload-pre-processing 26m Normal ScalingReplicaSet deployment/e2e-worker-w3-1f2e7e-openshift-default Scaled up replica set e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc from 1 to 2 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-1f2e7e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Normal SuccessfulCreate replicaset/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc Created pod: e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg 26m Normal AddedInterface pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Add eth0 [10.133.0.158/23] from ovn-kubernetes 26m Normal Created pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Container created 26m Normal Started pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Container started 26m Normal Pulled pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Startup probe error: Get "http://10.133.0.158:15021/healthz/ready": dial tcp 10.133.0.158:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Startup probe failed: Get "http://10.133.0.158:15021/healthz/ready": dial tcp 10.133.0.158:15021: connect: connection refused 26m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Startup probe error: Get "http://10.133.0.158:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Startup probe failed: Get "http://10.133.0.158:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg Stopping container istio-proxy 26m Normal Killing pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Stopping container istio-proxy 26m Warning ProbeError pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Readiness probe error: HTTP probe failed with statuscode: 503... 26m Warning ProbeError pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Readiness probe error: HTTP probe failed with statuscode: 503... 26m Warning Unhealthy pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s Readiness probe failed: HTTP probe failed with statuscode: 503 26m Warning Unhealthy pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6mzq6 Readiness probe failed: HTTP probe failed with statuscode: 503 26m Normal Killing pod/payload-processing-e2e-inf-b-dcdb23-56c7f9c647-blhhk Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Stopping container payload-pre-processing 25m Normal Killing pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Stopping container istio-proxy 25m Warning Unhealthy pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning ProbeError pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-9c6mk Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-gbst7 Stopping container istio-proxy 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-gbst7 25m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 9; reason: All metrics below target 25m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 10 to 9 25m Normal Killing pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Stopping container istio-proxy 25m Warning Unhealthy pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning ProbeError pod/e2e-inf-b-dcdb23-openshift-default-6df5cd9845-ktbx4 Readiness probe error: HTTP probe failed with statuscode: 503... 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-s4f8g 25m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58d898bd55 from 9 to 4 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-5pm4h Stopping container istio-proxy 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Stopping container istio-proxy 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Stopping container istio-proxy 25m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-5pm4h 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-qbnpn 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-qbnpn Stopping container istio-proxy 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-8xt57 25m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58d898bd55 Deleted pod: maas-default-gateway-openshift-default-58d898bd55-gp9f6 25m Normal Killing pod/maas-default-gateway-openshift-default-58d898bd55-8xt57 Stopping container istio-proxy 25m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g Readiness probe failed: HTTP probe failed with statuscode: 503 25m Normal Killing pod/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-worker-w0-6606d6-868d47b4ff-t9cm4 Stopping container payload-processing 25m Normal Killing pod/payload-processing-e2e-inf-a-50ae96-667dc45bb9-4gl5g Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Stopping container payload-pre-processing 24m Normal Killing pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Stopping container istio-proxy 24m Warning Unhealthy pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Stopping container payload-processing 24m Normal Killing pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-6f26n Stopping container istio-proxy 24m Normal Killing pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z Stopping container istio-proxy 24m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-eac348-7f8ff88f75-swtjh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning Unhealthy pod/payload-processing-e2e-inf-b-eac348-cfd67fd55-rbx6n Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6 Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-worker-w4-ffee69-5b5b46ddbf-xnfrp Stopping container payload-processing 23m Normal Killing pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z Stopping container istio-proxy 23m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58d898bd55 Created pod: maas-default-gateway-openshift-default-58d898bd55-4jgbz 23m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Add eth0 [10.132.0.125/23] from ovn-kubernetes 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Add eth0 [10.132.0.126/23] from ovn-kubernetes 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Add eth0 [10.133.0.161/23] from ovn-kubernetes 23m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 4 to 8 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Add eth0 [10.133.0.160/23] from ovn-kubernetes 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Container created 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Container created 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Container started 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Container started 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Container created 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Container started 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Container created 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Container started 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Startup probe failed: Get "http://10.133.0.160:15021/healthz/ready": dial tcp 10.133.0.160:15021: connect: connection refused 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Startup probe error: Get "http://10.133.0.160:15021/healthz/ready": dial tcp 10.133.0.160:15021: connect: connection refused... 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Startup probe error: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused... 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Startup probe error: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused... 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Startup probe failed: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Startup probe failed: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Startup probe failed: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Startup probe error: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused... 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Add eth0 [10.132.0.127/23] from ovn-kubernetes 23m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: 23m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58d898bd55 from 8 to 10 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Add eth0 [10.134.0.245/23] from ovn-kubernetes 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Container started 23m Normal Pulled pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Container created 23m Normal Created pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Container created 23m Normal Started pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Container started 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Startup probe error: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused... 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Startup probe failed: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused 23m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Startup probe error: Get "http://10.134.0.245:15021/healthz/ready": dial tcp 10.134.0.245:15021: connect: connection refused... 23m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Startup probe failed: Get "http://10.134.0.245:15021/healthz/ready": dial tcp 10.134.0.245:15021: connect: connection refused 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Readiness probe failed: Get "http://10.133.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl Readiness probe error: Get "http://10.133.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Readiness probe failed: Get "http://10.132.0.127:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-rdzr6 Readiness probe error: Get "http://10.132.0.127:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 16m Normal AddedInterface pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Add eth0 [10.133.0.163/23] from ovn-kubernetes 16m Normal SuccessfulCreate replicaset/e2e-worker-single-main-98d4f3-openshift-default-648c896b4f Created pod: e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz 16m Normal ScalingReplicaSet deployment/e2e-worker-single-main-98d4f3-openshift-default Scaled up replica set e2e-worker-single-main-98d4f3-openshift-default-648c896b4f from 0 to 1 16m Normal AddedLabel gateway/e2e-worker-single-main-98d4f3 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-98d4f3 15m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-98d4f3 Scaled up replica set payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb7dd5f from 0 to 1 15m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-98d4f3-6949b987c5 Error creating: pods "payload-processing-e2e-worker-single-main-98d4f3-6949b987c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3: serviceaccount "payload-processing-e2e-worker-single-main-98d4f3" not found 15m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-98d4f3 Scaled up replica set payload-processing-e2e-worker-single-main-98d4f3-6949b987c5 from 0 to 1 15m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-98d4f3-6949b987c5 Created pod: payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr 15m Normal Pulled pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb7dd5f Error creating: pods "payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb7dd5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-98d4f3: serviceaccount "payload-processing-e2e-worker-single-main-98d4f3" not found 15m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb7dd5f Created pod: payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d 15m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Add eth0 [10.134.1.4/23] from ovn-kubernetes 15m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Normal Started pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Container started 15m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Add eth0 [10.134.1.5/23] from ovn-kubernetes 15m Normal Created pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Container created 15m Normal Created pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Container created 15m Normal Started pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Container started 15m Normal Started pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Container started 15m Normal Created pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Container created 15m Normal Pulled pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 15m Warning Unhealthy pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Startup probe failed: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused 15m Warning ProbeError pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Startup probe error: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused... 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-98d4f3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 15m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-98d4f3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 15m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-98d4f3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-98d4f3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 14m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Stopping container payload-pre-processing 14m Normal Killing pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Stopping container payload-processing 14m Normal Killing pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz Stopping container istio-proxy 14m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-98d4f3-6b7cb22v7d Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-98d4f3-6949b987ct6rfr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 11m Normal AddedLabel gateway/e2e-worker-single-main-b435d8 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-b435d8 11m Normal ScalingReplicaSet deployment/e2e-worker-single-main-b435d8-openshift-default Scaled up replica set e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf from 0 to 1 11m Normal SuccessfulCreate replicaset/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf Created pod: e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd 11m Normal AddedInterface pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Add eth0 [10.133.0.165/23] from ovn-kubernetes 11m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-b435d8 Scaled up replica set payload-processing-e2e-worker-single-main-b435d8-68b5756987 from 0 to 1 11m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-b435d8-8599cff5dc Created pod: payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw 11m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-b435d8 Scaled up replica set payload-pre-processing-e2e-worker-single-main-b435d8-8599cff5dc from 0 to 1 11m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-b435d8-8599cff5dc Error creating: pods "payload-pre-processing-e2e-worker-single-main-b435d8-8599cff5dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-b435d8: serviceaccount "payload-processing-e2e-worker-single-main-b435d8" not found 11m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-b435d8-68b5756987 Created pod: payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm 11m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-b435d8-68b5756987 Error creating: pods "payload-processing-e2e-worker-single-main-b435d8-68b5756987-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-b435d8: serviceaccount "payload-processing-e2e-worker-single-main-b435d8" not found 11m Normal Started pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Container started 11m Normal Created pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Container created 11m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Add eth0 [10.134.1.12/23] from ovn-kubernetes 11m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal Started pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Container started 11m Normal Created pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Container created 11m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Add eth0 [10.134.1.13/23] from ovn-kubernetes 11m Normal Pulled pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-b435d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-b435d8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Normal Started pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Container started 10m Normal Pulled pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Normal Created pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Container created 10m Warning ProbeError pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Startup probe error: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused... 10m Warning Unhealthy pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Startup probe failed: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-b435d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-b435d8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m41s Normal Killing pod/payload-processing-e2e-worker-single-main-b435d8-68b575698v24jm Stopping container payload-processing 9m40s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-b435d8-8599cqlgvw Stopping container payload-pre-processing 9m22s Normal Killing pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Stopping container istio-proxy 9m17s Warning ProbeError pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Readiness probe error: HTTP probe failed with statuscode: 503... 9m17s Warning Unhealthy pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd Readiness probe failed: HTTP probe failed with statuscode: 503 7m54s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Readiness probe failed: Get "http://10.133.0.94:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m54s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Readiness probe failed: Get "http://10.134.0.139:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m54s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-kx6fw Readiness probe error: Get "http://10.134.0.139:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m54s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-qrckb Readiness probe error: Get "http://10.133.0.94:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m39s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Readiness probe error: Get "http://10.134.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m39s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-pmhtb Readiness probe failed: Get "http://10.134.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m39s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Readiness probe failed: Get "http://10.134.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m39s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-mrk4m Readiness probe error: Get "http://10.134.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m30s Normal AddedLabel gateway/e2e-ait-16cc64e2 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-16cc64e2 6m30s Normal ScalingReplicaSet deployment/e2e-ait-16cc64e2-openshift-default Scaled up replica set e2e-ait-16cc64e2-openshift-default-69f874c877 from 0 to 1 6m30s Normal SuccessfulCreate replicaset/e2e-ait-16cc64e2-openshift-default-69f874c877 Created pod: e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q 6m29s Normal AddedInterface pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Add eth0 [10.134.1.20/23] from ovn-kubernetes 6m23s Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7 Error creating: pods "payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-16cc64e2: serviceaccount "payload-processing-e2e-ait-16cc64e2" not found 6m23s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-16cc64e2 Scaled up replica set payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7 from 0 to 1 6m23s Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-16cc64e2 Scaled up replica set payload-processing-e2e-ait-16cc64e2-5664d9975f from 0 to 1 6m23s Warning FailedCreate replicaset/payload-processing-e2e-ait-16cc64e2-5664d9975f Error creating: pods "payload-processing-e2e-ait-16cc64e2-5664d9975f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-16cc64e2: serviceaccount "payload-processing-e2e-ait-16cc64e2" not found 6m22s Normal AddedInterface pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Add eth0 [10.133.0.169/23] from ovn-kubernetes 6m22s Normal Pulled pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m22s Normal Created pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Container created 6m22s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7 Created pod: payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw 6m22s Normal AddedInterface pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Add eth0 [10.133.0.168/23] from ovn-kubernetes 6m22s Normal Started pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Container started 6m22s Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-16cc64e2-5664d9975f Created pod: payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 6m21s Normal Started pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Container started 6m21s Normal Created pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Container created 6m21s Normal Pulled pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m2s Normal Started pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Container started 6m2s Normal Created pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Container created 6m2s Normal Pulled pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m59s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-16cc64e2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m59s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-16cc64e2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m56s Warning ProbeError pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Startup probe error: Get "http://10.134.1.20:15021/healthz/ready": dial tcp 10.134.1.20:15021: connect: connection refused... 5m56s Warning Unhealthy pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Startup probe failed: Get "http://10.134.1.20:15021/healthz/ready": dial tcp 10.134.1.20:15021: connect: connection refused 5m39s Normal Killing pod/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284 Stopping container payload-processing 5m38s Normal Killing pod/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw Stopping container payload-pre-processing 5m29s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-16cc64e2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m29s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-16cc64e2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m29s Normal Killing pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Stopping container istio-proxy 5m27s Normal SuccessfulCreate replicaset/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc Created pod: e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x 5m27s Normal AddedLabel gateway/e2e-authcfg-http-3ef065 Added label istio.io/rev=openshift-gateway to gateway e2e-authcfg-http-3ef065 5m27s Normal ScalingReplicaSet deployment/e2e-authcfg-http-3ef065-openshift-default Scaled up replica set e2e-authcfg-http-3ef065-openshift-default-644d5db5cc from 0 to 1 5m26s Normal AddedInterface pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Add eth0 [10.133.0.171/23] from ovn-kubernetes 5m25s Warning ProbeError pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Readiness probe error: HTTP probe failed with statuscode: 503... 5m25s Warning Unhealthy pod/e2e-ait-16cc64e2-openshift-default-69f874c877-mv66q Readiness probe failed: HTTP probe failed with statuscode: 503 5m20s Normal ScalingReplicaSet deployment/payload-processing-e2e-authcfg-http-3ef065 Scaled up replica set payload-processing-e2e-authcfg-http-3ef065-599699d4b5 from 0 to 1 5m20s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-authcfg-http-3ef065 Scaled up replica set payload-pre-processing-e2e-authcfg-http-3ef065-745b446567 from 0 to 1 5m19s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567 Created pod: payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 5m19s Normal SuccessfulCreate replicaset/payload-processing-e2e-authcfg-http-3ef065-599699d4b5 Created pod: payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 5m19s Warning FailedCreate replicaset/payload-processing-e2e-authcfg-http-3ef065-599699d4b5 Error creating: pods "payload-processing-e2e-authcfg-http-3ef065-599699d4b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-3ef065: serviceaccount "payload-processing-e2e-authcfg-http-3ef065" not found 5m19s Normal AddedInterface pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Add eth0 [10.134.1.22/23] from ovn-kubernetes 5m19s Warning FailedCreate replicaset/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567 Error creating: pods "payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-3ef065: serviceaccount "payload-processing-e2e-authcfg-http-3ef065" not found 5m18s Normal Started pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Container started 5m18s Normal Started pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Container started 5m18s Normal Pulled pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m18s Normal Pulled pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m18s Normal Created pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Container created 5m18s Normal AddedInterface pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Add eth0 [10.134.1.23/23] from ovn-kubernetes 5m18s Normal Created pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Container created 4m59s Normal Created pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Container created 4m59s Normal Pulled pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m59s Normal Started pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Container started 4m57s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-3ef065-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m57s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-3ef065-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m54s Warning ProbeError pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Startup probe error: Get "http://10.133.0.171:15021/healthz/ready": dial tcp 10.133.0.171:15021: connect: connection refused... 4m54s Warning Unhealthy pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Startup probe failed: Get "http://10.133.0.171:15021/healthz/ready": dial tcp 10.133.0.171:15021: connect: connection refused 4m11s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-3ef065-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m11s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-3ef065-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 3m26s Normal Killing pod/payload-pre-processing-e2e-authcfg-http-3ef065-745b446567-htxq5 Stopping container payload-pre-processing 3m26s Normal Killing pod/payload-processing-e2e-authcfg-http-3ef065-599699d4b5-j4h65 Stopping container payload-processing 3m9s Normal Killing pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Stopping container istio-proxy 3m8s Warning Unhealthy pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Readiness probe failed: HTTP probe failed with statuscode: 503 3m8s Warning ProbeError pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x Readiness probe error: HTTP probe failed with statuscode: 503... 2m44s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Readiness probe error: Get "http://10.134.0.245:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m44s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-7bv22 Readiness probe failed: Get "http://10.134.0.245:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m44s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Readiness probe failed: Get "http://10.132.0.125:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m44s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-4jgbz Readiness probe error: Get "http://10.132.0.125:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m44s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Readiness probe error: Get "http://10.133.0.161:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m44s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-f66wc Readiness probe failed: Get "http://10.133.0.161:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m44s Warning Unhealthy pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Readiness probe failed: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m44s Warning ProbeError pod/maas-default-gateway-openshift-default-58d898bd55-xqv52 Readiness probe error: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...