=== events in openshift-ingress (2026-10-30T21:33:17Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 31m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl to ip-10-0-130-195.us-west-1.compute.internal 45m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zd4sq to ip-10-0-141-29.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kwkj5 to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Successfully assigned openshift-ingress/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh to ip-10-0-141-29.us-west-1.compute.internal 63m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458 to ip-10-0-138-34.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Successfully assigned openshift-ingress/e2e-mt-27219c73-openshift-default-88d599849-rnbxz to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pdvns to ip-10-0-130-195.us-west-1.compute.internal 33m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pr86v to ip-10-0-130-195.us-west-1.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-sf6r9 to ip-10-0-141-29.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Successfully assigned openshift-ingress/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 to ip-10-0-141-29.us-west-1.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tjm7q to ip-10-0-141-29.us-west-1.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xbx2q to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-xjvz5 to ip-10-0-141-29.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jjg48 to ip-10-0-130-195.us-west-1.compute.internal 57m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hjtwx to ip-10-0-141-29.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Successfully assigned openshift-ingress/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk to ip-10-0-130-195.us-west-1.compute.internal 30m Normal Scheduled pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Successfully assigned openshift-ingress/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv to ip-10-0-130-195.us-west-1.compute.internal 36m Normal Scheduled pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Successfully assigned openshift-ingress/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 to ip-10-0-141-29.us-west-1.compute.internal 68m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fmmz4 to ip-10-0-141-29.us-west-1.compute.internal 68m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl to ip-10-0-141-29.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Successfully assigned openshift-ingress/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv to ip-10-0-141-29.us-west-1.compute.internal 63m Normal Scheduled pod/payload-pre-processing-7b98bb678c-9bl76 Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-9bl76 to ip-10-0-130-195.us-west-1.compute.internal 33m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fg56m to ip-10-0-141-29.us-west-1.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-f8t7j to ip-10-0-130-195.us-west-1.compute.internal 73m Normal Scheduled pod/router-default-67cdd9f98c-zf64k Successfully assigned openshift-ingress/router-default-67cdd9f98c-zf64k to ip-10-0-141-29.us-west-1.compute.internal 74m Warning FailedScheduling pod/router-default-67cdd9f98c-zf64k no nodes available to schedule pods 51m Normal Scheduled pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Successfully assigned openshift-ingress/e2e-ait-23cde229-openshift-default-659c875689-4fjrs to ip-10-0-130-195.us-west-1.compute.internal 79m Warning FailedScheduling pod/router-default-67cdd9f98c-zf64k no nodes available to schedule pods 52m Normal Scheduled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Successfully assigned openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j to ip-10-0-141-29.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Successfully assigned openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Successfully assigned openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp to ip-10-0-138-34.us-west-1.compute.internal 56m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh to ip-10-0-141-29.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb to ip-10-0-138-34.us-west-1.compute.internal 57m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 to ip-10-0-130-195.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 to ip-10-0-138-34.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr to ip-10-0-138-34.us-west-1.compute.internal 57m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt to ip-10-0-130-195.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Successfully assigned openshift-ingress/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg to ip-10-0-130-195.us-west-1.compute.internal 53m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj to ip-10-0-130-195.us-west-1.compute.internal 56m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c to ip-10-0-138-34.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Successfully assigned openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs to ip-10-0-141-29.us-west-1.compute.internal 53m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt to ip-10-0-130-195.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc to ip-10-0-130-195.us-west-1.compute.internal 53m Normal Scheduled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Successfully assigned openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw to ip-10-0-138-34.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng to ip-10-0-138-34.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj to ip-10-0-130-195.us-west-1.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d2bvn to ip-10-0-130-195.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 to ip-10-0-130-195.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s to ip-10-0-130-195.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Successfully assigned openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 to ip-10-0-141-29.us-west-1.compute.internal 6m25s Normal Scheduled pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Successfully assigned openshift-ingress/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Successfully assigned openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 to ip-10-0-138-34.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Successfully assigned openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g to ip-10-0-141-29.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd to ip-10-0-138-34.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw to ip-10-0-130-195.us-west-1.compute.internal 54m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp to ip-10-0-141-29.us-west-1.compute.internal 56m Normal Scheduled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Successfully assigned openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b to ip-10-0-141-29.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Successfully assigned openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv to ip-10-0-138-34.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Successfully assigned openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 to ip-10-0-130-195.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Successfully assigned openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m to ip-10-0-138-34.us-west-1.compute.internal 11m Normal Scheduled pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Successfully assigned openshift-ingress/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j to ip-10-0-130-195.us-west-1.compute.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx to ip-10-0-130-195.us-west-1.compute.internal 62m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc to ip-10-0-141-29.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 to ip-10-0-141-29.us-west-1.compute.internal 41m Normal Scheduled pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Successfully assigned openshift-ingress/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 to ip-10-0-130-195.us-west-1.compute.internal 15m Normal Scheduled pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Successfully assigned openshift-ingress/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt to ip-10-0-130-195.us-west-1.compute.internal 56m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw to ip-10-0-138-34.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw to ip-10-0-138-34.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt to ip-10-0-130-195.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb to ip-10-0-130-195.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Successfully assigned openshift-ingress/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf to ip-10-0-130-195.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx to ip-10-0-138-34.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n to ip-10-0-130-195.us-west-1.compute.internal 58m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr to ip-10-0-130-195.us-west-1.compute.internal 52m Normal Scheduled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Successfully assigned openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r to ip-10-0-130-195.us-west-1.compute.internal 42m Normal Scheduled pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Successfully assigned openshift-ingress/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml to ip-10-0-130-195.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Successfully assigned openshift-ingress/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h to ip-10-0-130-195.us-west-1.compute.internal 61m Normal Scheduled pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k to ip-10-0-130-195.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Successfully assigned openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb to ip-10-0-138-34.us-west-1.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj to ip-10-0-141-29.us-west-1.compute.internal 15m Normal Scheduled pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r to ip-10-0-141-29.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp to ip-10-0-141-29.us-west-1.compute.internal 42m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 to ip-10-0-138-34.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx to ip-10-0-138-34.us-west-1.compute.internal 45m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh to ip-10-0-141-29.us-west-1.compute.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 to ip-10-0-141-29.us-west-1.compute.internal 45m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 to ip-10-0-141-29.us-west-1.compute.internal 45m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl to ip-10-0-141-29.us-west-1.compute.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng to ip-10-0-138-34.us-west-1.compute.internal 47m Normal Scheduled pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Successfully assigned openshift-ingress/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg to ip-10-0-141-29.us-west-1.compute.internal 46m Normal Scheduled pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Successfully assigned openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh to ip-10-0-141-29.us-west-1.compute.internal 39m Normal Scheduled pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Successfully assigned openshift-ingress/payload-processing-e2e-mt-27219c73-57f8548544-drklv to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Successfully assigned openshift-ingress/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz to ip-10-0-141-29.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Successfully assigned openshift-ingress/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j to ip-10-0-130-195.us-west-1.compute.internal 30m Normal Scheduled pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm to ip-10-0-141-29.us-west-1.compute.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh to ip-10-0-141-29.us-west-1.compute.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q to ip-10-0-141-29.us-west-1.compute.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws to ip-10-0-141-29.us-west-1.compute.internal 31m Normal Scheduled pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 to ip-10-0-130-195.us-west-1.compute.internal 48m Normal Scheduled pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Successfully assigned openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf to ip-10-0-138-34.us-west-1.compute.internal 5m13s Normal Scheduled pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Successfully assigned openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v to ip-10-0-141-29.us-west-1.compute.internal 34m Normal Scheduled pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 to ip-10-0-141-29.us-west-1.compute.internal 38m Normal Scheduled pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Successfully assigned openshift-ingress/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw to ip-10-0-141-29.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Successfully assigned openshift-ingress/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx to ip-10-0-141-29.us-west-1.compute.internal 36m Normal Scheduled pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Successfully assigned openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg to ip-10-0-138-34.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Successfully assigned openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc to ip-10-0-138-34.us-west-1.compute.internal 37m Normal Scheduled pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Successfully assigned openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz to ip-10-0-138-34.us-west-1.compute.internal 41m Normal Scheduled pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Successfully assigned openshift-ingress/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h to ip-10-0-141-29.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Successfully assigned openshift-ingress/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 to ip-10-0-141-29.us-west-1.compute.internal 6m18s Normal Scheduled pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Successfully assigned openshift-ingress/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Successfully assigned openshift-ingress/payload-processing-e2e-ait-8660b337-f578c795b-z45vn to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Successfully assigned openshift-ingress/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x to ip-10-0-130-195.us-west-1.compute.internal 48m Normal Scheduled pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Successfully assigned openshift-ingress/payload-processing-e2e-ait-0d757729-7857666684-6pxgx to ip-10-0-141-29.us-west-1.compute.internal 69m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-lc888 Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888 to ip-10-0-138-34.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Successfully assigned openshift-ingress/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p to ip-10-0-130-195.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg to ip-10-0-138-34.us-west-1.compute.internal 33m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-b6sv7 to ip-10-0-141-29.us-west-1.compute.internal 63m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8l6hk to ip-10-0-141-29.us-west-1.compute.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 to ip-10-0-141-29.us-west-1.compute.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr to ip-10-0-130-195.us-west-1.compute.internal 63m Normal Scheduled pod/payload-processing-5f78687d5b-vtz7b Successfully assigned openshift-ingress/payload-processing-5f78687d5b-vtz7b to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p to ip-10-0-130-195.us-west-1.compute.internal 6m18s Normal Scheduled pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv to ip-10-0-141-29.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Successfully assigned openshift-ingress/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp to ip-10-0-130-195.us-west-1.compute.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm to ip-10-0-141-29.us-west-1.compute.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf to ip-10-0-138-34.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg to ip-10-0-138-34.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Successfully assigned openshift-ingress/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t to ip-10-0-130-195.us-west-1.compute.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg to ip-10-0-141-29.us-west-1.compute.internal 66m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-9rshc Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc to ip-10-0-138-34.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf to ip-10-0-141-29.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg to ip-10-0-141-29.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Successfully assigned openshift-ingress/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 to ip-10-0-141-29.us-west-1.compute.internal 67m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp to ip-10-0-141-29.us-west-1.compute.internal 42m Normal Scheduled pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Successfully assigned openshift-ingress/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pzcsw to ip-10-0-130-195.us-west-1.compute.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l to ip-10-0-141-29.us-west-1.compute.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 to ip-10-0-141-29.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Successfully assigned openshift-ingress/e2e-api-b-65948c-openshift-default-b44f97585-gz54k to ip-10-0-141-29.us-west-1.compute.internal 5m13s Normal Scheduled pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Successfully assigned openshift-ingress/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl to ip-10-0-141-29.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr to ip-10-0-141-29.us-west-1.compute.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h to ip-10-0-138-34.us-west-1.compute.internal 57m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-mr976 to ip-10-0-141-29.us-west-1.compute.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh to ip-10-0-138-34.us-west-1.compute.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql to ip-10-0-141-29.us-west-1.compute.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q to ip-10-0-141-29.us-west-1.compute.internal 30m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw to ip-10-0-141-29.us-west-1.compute.internal 31m Normal Scheduled pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Successfully assigned openshift-ingress/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj to ip-10-0-130-195.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j to ip-10-0-141-29.us-west-1.compute.internal 67m Normal Scheduled pod/kube-auth-proxy-557f979cb4-ctpdv Successfully assigned openshift-ingress/kube-auth-proxy-557f979cb4-ctpdv to ip-10-0-130-195.us-west-1.compute.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg to ip-10-0-141-29.us-west-1.compute.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 to ip-10-0-141-29.us-west-1.compute.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb to ip-10-0-141-29.us-west-1.compute.internal 34m Normal Scheduled pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Successfully assigned openshift-ingress/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj to ip-10-0-141-29.us-west-1.compute.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l to ip-10-0-138-34.us-west-1.compute.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq to ip-10-0-141-29.us-west-1.compute.internal 67m Normal Scheduled pod/kube-auth-proxy-557f979cb4-v95bs Successfully assigned openshift-ingress/kube-auth-proxy-557f979cb4-v95bs to ip-10-0-138-34.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 to ip-10-0-141-29.us-west-1.compute.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 to ip-10-0-141-29.us-west-1.compute.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n to ip-10-0-141-29.us-west-1.compute.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp to ip-10-0-141-29.us-west-1.compute.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt to ip-10-0-138-34.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 to ip-10-0-138-34.us-west-1.compute.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz to ip-10-0-138-34.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 to ip-10-0-138-34.us-west-1.compute.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl to ip-10-0-130-195.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Successfully assigned openshift-ingress/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s to ip-10-0-130-195.us-west-1.compute.internal 15m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg to ip-10-0-141-29.us-west-1.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh to ip-10-0-141-29.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Successfully assigned openshift-ingress/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v to ip-10-0-141-29.us-west-1.compute.internal 5m21s Normal Scheduled pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Successfully assigned openshift-ingress/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn to ip-10-0-130-195.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll to ip-10-0-138-34.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2qgzh to ip-10-0-141-29.us-west-1.compute.internal 61m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn to ip-10-0-130-195.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-5bb9c to ip-10-0-141-29.us-west-1.compute.internal 50m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-695mh Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh to ip-10-0-138-34.us-west-1.compute.internal 63m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7cs6l to ip-10-0-130-195.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Successfully assigned openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm to ip-10-0-138-34.us-west-1.compute.internal 80m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-67cdd9f98c from 0 to 1 80m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 80m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 80m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 79m Normal SuccessfulCreate replicaset/router-default-67cdd9f98c Created pod: router-default-67cdd9f98c-zf64k 74m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 73m Warning FailedMount pod/router-default-67cdd9f98c-zf64k MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 73m Warning FailedMount pod/router-default-67cdd9f98c-zf64k MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 73m Warning FailedMount pod/router-default-67cdd9f98c-zf64k MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 73m Warning FailedMount pod/router-default-67cdd9f98c-zf64k MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 73m Normal AddedInterface pod/router-default-67cdd9f98c-zf64k Add eth0 [10.132.0.10/23] from ovn-kubernetes 73m Normal Started pod/router-default-67cdd9f98c-zf64k Container started 73m Normal Pulled pod/router-default-67cdd9f98c-zf64k Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 73m Normal Created pod/router-default-67cdd9f98c-zf64k Container created 69m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 69m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 69m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-lc888 69m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-lc888 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 69m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-lc888 Add eth0 [10.133.0.17/23] from ovn-kubernetes 68m Normal Created pod/istiod-openshift-gateway-7876bfcb74-lc888 Container created 68m Normal Started pod/istiod-openshift-gateway-7876bfcb74-lc888 Container started 68m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-lc888 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.09s (4.09s including waiting). Image size: 185476477 bytes. 68m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-lc888 Readiness probe failed: HTTP probe failed with statuscode: 503 68m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-lc888 Readiness probe error: HTTP probe failed with statuscode: 503... 68m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-55f56bc75f from 0 to 1 68m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-55f56bc75f Created pod: openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl 68m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Add eth0 [10.132.0.33/23] from ovn-kubernetes 68m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-fmmz4 68m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 0 to 1 68m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 68m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Add eth0 [10.132.0.34/23] from ovn-kubernetes 68m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 68m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 68m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 68m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.672s (8.672s including waiting). Image size: 183555333 bytes. 68m Normal Started pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Container started 68m Normal Created pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Container created 68m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-82pfl Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 11.388s (11.388s including waiting). Image size: 183555333 bytes. 68m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 68m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 68m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 68m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Normal SuccessfulCreate replicaset/kube-auth-proxy-557f979cb4 Created pod: kube-auth-proxy-557f979cb4-ctpdv 67m Normal SuccessfulCreate replicaset/kube-auth-proxy-557f979cb4 Created pod: kube-auth-proxy-557f979cb4-v95bs 67m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-557f979cb4 from 0 to 2 67m Normal Pulled pod/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 67m Normal AddedInterface pod/kube-auth-proxy-557f979cb4-v95bs Add eth0 [10.133.0.20/23] from ovn-kubernetes 67m Normal Started pod/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp Container started 67m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-7845f94dd5 from 0 to 1 67m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp Add eth0 [10.132.0.41/23] from ovn-kubernetes 67m Normal Created pod/data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp Container created 67m Normal Pulling pod/kube-auth-proxy-557f979cb4-v95bs Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 67m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-7845f94dd5 Created pod: data-science-gateway-data-science-gateway-class-7845f94dd5r8mmp 67m Normal AddedInterface pod/kube-auth-proxy-557f979cb4-ctpdv Add eth0 [10.134.0.31/23] from ovn-kubernetes 67m Normal Pulling pod/kube-auth-proxy-557f979cb4-ctpdv Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 67m Normal Started pod/kube-auth-proxy-557f979cb4-v95bs Container started 67m Normal Started pod/kube-auth-proxy-557f979cb4-ctpdv Container started 67m Normal Created pod/kube-auth-proxy-557f979cb4-ctpdv Container created 67m Normal Pulled pod/kube-auth-proxy-557f979cb4-v95bs Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.055s (3.055s including waiting). Image size: 188904677 bytes. 67m Normal Created pod/kube-auth-proxy-557f979cb4-v95bs Container created 67m Normal Pulled pod/kube-auth-proxy-557f979cb4-ctpdv Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.853s (2.853s including waiting). Image size: 188904677 bytes. 67m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 67m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 67m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 66m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-9rshc 66m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-9rshc Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 66m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-9rshc Add eth0 [10.133.0.25/23] from ovn-kubernetes 66m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-9rshc Container created 66m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-9rshc Container started 66m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-lc888 66m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-lc888 Stopping container discovery 66m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 66m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-lc888 Readiness probe failed: Get "http://10.133.0.17:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 66m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-lc888 Readiness probe error: Get "http://10.133.0.17:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 66m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 66m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Readiness probe error: Get "http://10.132.0.33:15021/healthz/ready": read tcp 10.132.0.2:54708->10.132.0.33:15021: read: connection reset by peer... 65m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Container created 65m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Readiness probe failed: Get "http://10.132.0.33:15021/healthz/ready": read tcp 10.132.0.2:54708->10.132.0.33:15021: read: connection reset by peer 65m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 65m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Container started 65m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Startup probe failed: Get "http://10.132.0.33:15021/healthz/ready": dial tcp 10.132.0.33:15021: connect: connection refused 65m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Startup probe error: Get "http://10.132.0.33:15021/healthz/ready": dial tcp 10.132.0.33:15021: connect: connection refused... 65m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 63m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-7cs6l 63m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 63m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 1 to 2 63m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 63m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Add eth0 [10.134.0.39/23] from ovn-kubernetes 63m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Container created 63m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Container started 63m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.489s (3.489s including waiting). Image size: 183555333 bytes. 63m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 63m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 63m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-8l6hk 63m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-k7458 63m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Container created 63m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 63m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Add eth0 [10.132.0.52/23] from ovn-kubernetes 63m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Container started 63m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 63m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Add eth0 [10.133.0.31/23] from ovn-kubernetes 63m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.989s (3.989s including waiting). Image size: 183555333 bytes. 63m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Container created 63m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Container started 63m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused 63m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Startup probe error: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused... 63m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused... 63m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused 63m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 63m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-9bl76 63m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5f78687d5b from 0 to 1 63m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 63m Warning FailedCreate replicaset/payload-processing-5f78687d5b Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 63m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-9bl76 Add eth0 [10.134.0.43/23] from ovn-kubernetes 63m Normal Pulling pod/payload-processing-5f78687d5b-vtz7b Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 63m Normal AddedInterface pod/payload-processing-5f78687d5b-vtz7b Add eth0 [10.134.0.44/23] from ovn-kubernetes 63m Normal Pulling pod/payload-pre-processing-7b98bb678c-9bl76 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 63m Normal SuccessfulCreate replicaset/payload-processing-5f78687d5b Created pod: payload-processing-5f78687d5b-vtz7b 63m Normal Pulled pod/payload-processing-5f78687d5b-vtz7b Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.696s (2.696s including waiting). Image size: 125353965 bytes. 63m Normal Started pod/payload-pre-processing-7b98bb678c-9bl76 Container started 63m Normal Started pod/payload-processing-5f78687d5b-vtz7b Container started 63m Normal Created pod/payload-pre-processing-7b98bb678c-9bl76 Container created 63m Normal Pulled pod/payload-pre-processing-7b98bb678c-9bl76 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.774s (2.774s including waiting). Image size: 125353965 bytes. 63m Normal Created pod/payload-processing-5f78687d5b-vtz7b Container created 62m Normal AddedLabel gateway/e2e-worker-w6-4ee72d Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-4ee72d 62m Normal AddedLabel gateway/e2e-worker-w3-ae2d38 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-ae2d38 62m Normal AddedLabel gateway/e2e-worker-w0-0a78ac Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-0a78ac 62m Normal AddedLabel gateway/e2e-worker-w2-df031b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-df031b 62m Normal AddedLabel gateway/e2e-worker-w4-8e2fc1 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-8e2fc1 62m Normal AddedLabel gateway/e2e-worker-w1-5f9b95 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-5f9b95 62m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled up replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 0 to 1 62m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj 62m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled up replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 0 to 1 62m Normal AddedLabel gateway/e2e-worker-w5-a78bf4 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-a78bf4 62m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc 62m Normal SuccessfulCreate replicaset/e2e-worker-w2-df031b-openshift-default-77cd44ff85 Created pod: e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 62m Normal AddedInterface pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Add eth0 [10.134.0.46/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/e2e-worker-w2-df031b-openshift-default Scaled up replica set e2e-worker-w2-df031b-openshift-default-77cd44ff85 from 0 to 1 62m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Add eth0 [10.134.0.47/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/e2e-worker-w4-8e2fc1-openshift-default Scaled up replica set e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f from 0 to 1 62m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf 62m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn 62m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Add eth0 [10.132.0.55/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled up replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 0 to 1 62m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Add eth0 [10.132.0.54/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f Created pod: e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg 62m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 0 to 1 62m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Add eth0 [10.132.0.56/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc Created pod: e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 62m Normal AddedInterface pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Add eth0 [10.134.0.49/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/e2e-worker-w5-a78bf4-openshift-default Scaled up replica set e2e-worker-w5-a78bf4-openshift-default-79b556ccdc from 0 to 1 62m Normal AddedInterface pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Add eth0 [10.134.0.48/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-df031b Scaled up replica set payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759 from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-df031b Scaled up replica set payload-processing-e2e-worker-w2-df031b-7f46bd6f58 from 0 to 1 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-df031b-7f46bd6f58 Error creating: pods "payload-processing-e2e-worker-w2-df031b-7f46bd6f58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-df031b: serviceaccount "payload-processing-e2e-worker-w2-df031b" not found 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Add eth0 [10.134.0.50/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-df031b-7f46bd6f58 Created pod: payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759 Created pod: payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759 Error creating: pods "payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-df031b: serviceaccount "payload-processing-e2e-worker-w2-df031b" not found 62m Normal AddedInterface pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Add eth0 [10.134.0.51/23] from ovn-kubernetes 62m Normal Pulled pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal Created pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Container created 62m Normal Started pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Container started 62m Normal Started pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Container started 62m Normal Created pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Container created 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-4ee72d Scaled up replica set payload-processing-e2e-worker-w6-4ee72d-766db79cc6 from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-4ee72d Scaled up replica set payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787 from 0 to 1 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-4ee72d-766db79cc6 Error creating: pods "payload-processing-e2e-worker-w6-4ee72d-766db79cc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-4ee72d: serviceaccount "payload-processing-e2e-worker-w6-4ee72d" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787 Created pod: payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787 Error creating: pods "payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-4ee72d: serviceaccount "payload-processing-e2e-worker-w6-4ee72d" not found 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-4ee72d-766db79cc6 Created pod: payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Add eth0 [10.132.0.57/23] from ovn-kubernetes 61m Normal Pulling pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 61m Normal Created pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Container created 61m Normal Started pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Container started 61m Normal AddedInterface pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Add eth0 [10.134.0.52/23] from ovn-kubernetes 61m Normal Pulled pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46 Created pod: payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-0a78ac Scaled up replica set payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46 from 0 to 1 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46 Error creating: pods "payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-0a78ac: serviceaccount "payload-processing-e2e-worker-w0-0a78ac" not found 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-0a78ac Scaled up replica set payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59 from 0 to 1 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59 Error creating: pods "payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-0a78ac: serviceaccount "payload-processing-e2e-worker-w0-0a78ac" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59 Created pod: payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt 61m Normal Created pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Container created 61m Normal Created pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Container created 61m Normal Pulled pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Add eth0 [10.134.0.54/23] from ovn-kubernetes 61m Normal Started pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Container started 61m Normal Started pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Container started 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Add eth0 [10.134.0.53/23] from ovn-kubernetes 61m Normal Started pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Container started 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.868s (2.868s including waiting). Image size: 125353965 bytes. 61m Normal Created pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Container created 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-a78bf4-857fb69748 Error creating: pods "payload-processing-e2e-worker-w5-a78bf4-857fb69748-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-a78bf4: serviceaccount "payload-processing-e2e-worker-w5-a78bf4" not found 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-a78bf4 Scaled up replica set payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c Created pod: payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c Error creating: pods "payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-a78bf4: serviceaccount "payload-processing-e2e-worker-w5-a78bf4" not found 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-a78bf4 Scaled up replica set payload-processing-e2e-worker-w5-a78bf4-857fb69748 from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-a78bf4-857fb69748 Created pod: payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw 61m Normal Pulled pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Container started 61m Normal Created pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Container created 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Add eth0 [10.132.0.59/23] from ovn-kubernetes 61m Normal AddedInterface pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Add eth0 [10.132.0.60/23] from ovn-kubernetes 61m Normal Created pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Container created 61m Normal Started pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Container started 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-5f9b95 Scaled up replica set payload-processing-e2e-worker-w1-5f9b95-56c969bbb8 from 0 to 1 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff Error creating: pods "payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-5f9b95: serviceaccount "payload-processing-e2e-worker-w1-5f9b95" not found 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-5f9b95 Scaled up replica set payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff from 0 to 1 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8 Created pod: payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8 Error creating: pods "payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-5f9b95: serviceaccount "payload-processing-e2e-worker-w1-5f9b95" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff Created pod: payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Started pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Container started 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Add eth0 [10.134.0.55/23] from ovn-kubernetes 61m Normal Created pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Container created 61m Normal Created pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Container created 61m Normal AddedInterface pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Add eth0 [10.134.0.56/23] from ovn-kubernetes 61m Normal Started pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Container started 61m Normal Pulled pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Container started 61m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Container created 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784 Created pod: payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784 Error creating: pods "payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1: serviceaccount "payload-processing-e2e-worker-w4-8e2fc1" not found 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-8e2fc1 Scaled up replica set payload-processing-e2e-worker-w4-8e2fc1-87b4cd784 from 0 to 1 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7 Error creating: pods "payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-8e2fc1: serviceaccount "payload-processing-e2e-worker-w4-8e2fc1" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7 Created pod: payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-8e2fc1 Scaled up replica set payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7 from 0 to 1 61m Normal Started pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Container started 61m Normal Started pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Container started 61m Normal Created pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Container created 61m Normal Created pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Container created 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Add eth0 [10.132.0.61/23] from ovn-kubernetes 61m Normal Pulled pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Add eth0 [10.134.0.57/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-ae2d38 Scaled up replica set payload-processing-e2e-worker-w3-ae2d38-d5c76b6db from 0 to 1 61m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-ae2d38 Scaled up replica set payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c from 0 to 1 61m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db Error creating: pods "payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-ae2d38: serviceaccount "payload-processing-e2e-worker-w3-ae2d38" not found 61m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c Created pod: payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 61m Normal AddedInterface pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Add eth0 [10.134.0.58/23] from ovn-kubernetes 61m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db Created pod: payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw 61m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c Error creating: pods "payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-ae2d38: serviceaccount "payload-processing-e2e-worker-w3-ae2d38" not found 61m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Add eth0 [10.132.0.62/23] from ovn-kubernetes 61m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Startup probe failed: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Startup probe error: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused... 61m Normal Started pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Container started 61m Normal Created pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Container created 61m Normal Created pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Container created 61m Normal Started pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Container started 61m Normal Pulled pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 61m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Container started 61m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Container created 61m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Container started 61m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Container created 61m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Startup probe error: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Startup probe failed: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused... 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal Pulled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Container started 61m Normal Created pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Container created 61m Warning Unhealthy pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Startup probe error: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused... 61m Normal Started pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Container started 61m Normal Pulled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Created pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Container created 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning Unhealthy pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused... 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Readiness probe failed: Get "http://10.132.0.56:15021/healthz/ready": read tcp 10.132.0.2:44258->10.132.0.56:15021: read: connection reset by peer 61m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Readiness probe error: Get "http://10.132.0.56:15021/healthz/ready": read tcp 10.132.0.2:44258->10.132.0.56:15021: read: connection reset by peer... 61m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Container created 61m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Container started 61m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused... 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Normal Pulled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Started pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Container started 60m Normal Created pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Container created 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Startup probe error: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Startup probe failed: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Container started 58m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Container created 58m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr 58m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled up replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 1 to 2 58m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 58m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Add eth0 [10.134.0.62/23] from ovn-kubernetes 58m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 57m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Add eth0 [10.134.0.67/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt 57m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 1 to 2 57m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled up replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 1 to 2 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Container started 57m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Container created 57m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Container created 57m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Add eth0 [10.134.0.68/23] from ovn-kubernetes 57m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Container started 57m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 57m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Startup probe error: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Startup probe failed: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused 57m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-8l6hk Stopping container istio-proxy 57m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 57m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-fmmz4 57m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 4 to 2 57m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-fmmz4 Stopping container istio-proxy 57m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-8l6hk 57m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 57m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-mr976 57m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 2 to 4 57m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-hjtwx 57m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Add eth0 [10.132.0.70/23] from ovn-kubernetes 57m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Container created 57m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Add eth0 [10.132.0.71/23] from ovn-kubernetes 57m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Container created 57m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Container started 57m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Container started 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 56m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 56m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Add eth0 [10.133.0.42/23] from ovn-kubernetes 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled up replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 2 to 3 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w5-a78bf4-openshift-default Scaled up replica set e2e-worker-w5-a78bf4-openshift-default-79b556ccdc from 1 to 2 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc Created pod: e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv 56m Normal Created pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Container created 56m Normal Pulled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Container started 56m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Container created 56m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Container started 56m Normal AddedInterface pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Add eth0 [10.132.0.72/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-ctppg 56m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 4 to 6 56m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-jjg48 56m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Add eth0 [10.133.0.43/23] from ovn-kubernetes 56m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Container created 56m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Container started 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Add eth0 [10.134.0.85/23] from ovn-kubernetes 56m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Container started 56m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Container created 56m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused... 56m Warning ProbeError pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Startup probe error: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Startup probe error: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Startup probe failed: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused 56m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 2 to 4 56m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh 56m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Container started 56m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Container created 56m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Add eth0 [10.133.0.44/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Add eth0 [10.132.0.73/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Container started 56m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Container created 56m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Startup probe error: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused... 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 4 to 5 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc 55m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Container created 55m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Add eth0 [10.134.0.91/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Container started 55m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused... 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled up replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 2 to 3 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Add eth0 [10.133.0.46/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Container created 55m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Container started 55m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled up replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 1 to 2 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp 54m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Container created 54m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Add eth0 [10.132.0.78/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Container started 54m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 5 to 8 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 53m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Container started 53m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Add eth0 [10.132.0.80/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Container created 53m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Container started 53m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Add eth0 [10.134.0.99/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Add eth0 [10.133.0.49/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Container created 53m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Container created 53m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Container started 53m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Readiness probe failed: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Readiness probe error: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Startup probe failed: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Startup probe error: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w2-df031b-openshift-default Scaled up replica set e2e-worker-w2-df031b-openshift-default-77cd44ff85 from 1 to 2 52m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff 52m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv 52m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled up replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 3 to 6 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r 52m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled up replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 2 to 4 52m Normal SuccessfulCreate replicaset/e2e-worker-w2-df031b-openshift-default-77cd44ff85 Created pod: e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 52m Normal Started pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Container started 52m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Container created 52m Normal AddedInterface pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Add eth0 [10.132.0.82/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Container started 52m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Container started 52m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Container created 52m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Add eth0 [10.133.0.50/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Add eth0 [10.132.0.83/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Container created 52m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Container started 52m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Add eth0 [10.133.0.51/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Container created 52m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Container started 52m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Add eth0 [10.134.0.104/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Add eth0 [10.134.0.105/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Container created 52m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Container created 52m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Container started 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w4-8e2fc1-openshift-default Scaled up replica set e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f from 1 to 2 52m Normal SuccessfulCreate replicaset/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f Created pod: e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g 52m Normal AddedInterface pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Add eth0 [10.132.0.84/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Container created 52m Normal Started pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Container started 52m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Startup probe failed: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Startup probe error: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Startup probe error: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Startup probe failed: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Startup probe failed: Get "http://10.134.0.104:15021/healthz/ready": dial tcp 10.134.0.104:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Startup probe error: Get "http://10.134.0.104:15021/healthz/ready": dial tcp 10.134.0.104:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Normal ScalingReplicaSet deployment/e2e-worker-w5-a78bf4-openshift-default Scaled up replica set e2e-worker-w5-a78bf4-openshift-default-79b556ccdc from 2 to 3 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc Created pod: e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m 52m Normal Started pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Container started 52m Normal Created pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Container created 52m Normal Pulled pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Add eth0 [10.133.0.52/23] from ovn-kubernetes 52m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 6 to 5 52m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-hjtwx Stopping container istio-proxy 52m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-hjtwx 52m Warning Unhealthy pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused... 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-df031b-openshift-default Scaled up replica set e2e-worker-w2-df031b-openshift-default-77cd44ff85 from 2 to 3 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-df031b-openshift-default-77cd44ff85 Created pod: e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp 51m Normal AddedInterface pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Add eth0 [10.133.0.54/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Container created 51m Normal Started pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Container started 51m Normal SuccessfulCreate replicaset/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df Created pod: e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh 51m Normal ScalingReplicaSet deployment/e2e-migrate-7f0ceebd-gw-openshift-default Scaled up replica set e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df from 0 to 1 51m Normal SuccessfulCreate replicaset/e2e-ait-8660b337-openshift-default-5b69bc746c Created pod: e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg 51m Normal ScalingReplicaSet deployment/e2e-ait-8660b337-openshift-default Scaled up replica set e2e-ait-8660b337-openshift-default-5b69bc746c from 0 to 1 51m Normal AddedLabel gateway/e2e-ait-8660b337 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-8660b337 51m Normal AddedLabel gateway/e2e-migrate-7f0ceebd-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-7f0ceebd-gw 51m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Add eth0 [10.132.0.87/23] from ovn-kubernetes 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs 51m Normal AddedInterface pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Add eth0 [10.134.0.127/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Created pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f Created pod: e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 51m Normal ScalingReplicaSet deployment/e2e-worker-w4-8e2fc1-openshift-default Scaled up replica set e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f from 2 to 3 51m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled up replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 3 to 6 51m Normal AddedInterface pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Add eth0 [10.132.0.86/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Add eth0 [10.134.0.128/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Container started 51m Normal Started pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Container started 51m Normal Created pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Container created 51m Normal Pulled pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Container created 51m Normal Pulled pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Container started 51m Normal AddedInterface pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Add eth0 [10.133.0.56/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Add eth0 [10.133.0.55/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Container started 51m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Container created 51m Normal Created pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Container created 51m Warning ProbeError pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Startup probe error: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Startup probe failed: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused 51m Normal ScalingReplicaSet deployment/e2e-ait-23cde229-openshift-default Scaled up replica set e2e-ait-23cde229-openshift-default-659c875689 from 0 to 1 51m Normal SuccessfulCreate replicaset/e2e-ait-23cde229-openshift-default-659c875689 Created pod: e2e-ait-23cde229-openshift-default-659c875689-4fjrs 51m Normal AddedLabel gateway/e2e-ait-23cde229 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-23cde229 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-7f0ceebd Scaled up replica set payload-processing-e2e-migrate-7f0ceebd-d6ff68448 from 0 to 1 51m Normal AddedInterface pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Add eth0 [10.134.0.129/23] from ovn-kubernetes 51m Warning ProbeError pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Startup probe error: Get "http://10.133.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Startup probe failed: Get "http://10.133.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Startup probe error: Get "http://10.134.0.128:15021/healthz/ready": dial tcp 10.134.0.128:15021: connect: connection refused... 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-7f0ceebd Scaled up replica set payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7 from 0 to 1 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7 Created pod: payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-7f0ceebd-d6ff68448 Created pod: payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7 Error creating: pods "payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-7f0ceebd: serviceaccount "payload-processing-e2e-migrate-7f0ceebd" not found 51m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 51m Warning FailedCreate replicaset/payload-processing-e2e-migrate-7f0ceebd-d6ff68448 Error creating: pods "payload-processing-e2e-migrate-7f0ceebd-d6ff68448-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-7f0ceebd: serviceaccount "payload-processing-e2e-migrate-7f0ceebd" not found 51m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Startup probe failed: Get "http://10.134.0.128:15021/healthz/ready": dial tcp 10.134.0.128:15021: connect: connection refused 51m Normal Started pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Container started 51m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Startup probe error: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused... 51m Normal Created pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Container created 51m Normal AddedInterface pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Add eth0 [10.132.0.89/23] from ovn-kubernetes 51m Normal Pulled pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Container created 51m Normal Started pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Container started 51m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Add eth0 [10.132.0.88/23] from ovn-kubernetes 51m Normal Pulled pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Startup probe failed: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused... 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-8660b337 Scaled up replica set payload-processing-e2e-ait-8660b337-f578c795b from 0 to 1 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f Error creating: pods "payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-8660b337: serviceaccount "payload-processing-e2e-ait-8660b337" not found 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-8660b337 Scaled up replica set payload-pre-processing-e2e-ait-8660b337-6b9886ff7f from 0 to 1 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled up replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 4 to 6 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled up replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 6 to 10 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f Created pod: payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 (combined from similar events): Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt 51m Warning FailedCreate replicaset/payload-processing-e2e-ait-8660b337-f578c795b Error creating: pods "payload-processing-e2e-ait-8660b337-f578c795b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-8660b337: serviceaccount "payload-processing-e2e-ait-8660b337" not found 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Created pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-8660b337-f578c795b Created pod: payload-processing-e2e-ait-8660b337-f578c795b-z45vn 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b 51m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Container created 51m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Container started 51m Normal Started pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Container started 51m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Container started 51m Normal Created pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Container created 51m Normal Pulled pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Add eth0 [10.134.0.133/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Add eth0 [10.132.0.91/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Add eth0 [10.132.0.90/23] from ovn-kubernetes 51m Normal Started pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Container started 51m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Container created 51m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Container started 51m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Add eth0 [10.133.0.57/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Container started 51m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Container created 51m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Add eth0 [10.133.0.58/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Container created 51m Normal Created pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Container created 51m Normal Pulled pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Add eth0 [10.132.0.92/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Container started 51m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Add eth0 [10.134.0.132/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Container created 51m Normal Started pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Container started 51m Normal AddedInterface pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Add eth0 [10.134.0.131/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Container created 51m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled up replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 8 to 10 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c (combined from similar events): Created pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf 51m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Container started 51m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Add eth0 [10.132.0.93/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Add eth0 [10.133.0.59/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Container created 51m Normal Pulled pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-23cde229 Scaled up replica set payload-pre-processing-e2e-ait-23cde229-7646fc45dc from 0 to 1 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-23cde229 Scaled up replica set payload-processing-e2e-ait-23cde229-7498b68f66 from 0 to 1 51m Warning FailedCreate replicaset/payload-processing-e2e-ait-23cde229-7498b68f66 Error creating: pods "payload-processing-e2e-ait-23cde229-7498b68f66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-23cde229: serviceaccount "payload-processing-e2e-ait-23cde229" not found 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-23cde229-7646fc45dc Error creating: pods "payload-pre-processing-e2e-ait-23cde229-7646fc45dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-23cde229: serviceaccount "payload-processing-e2e-ait-23cde229" not found 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-23cde229-7498b68f66 Created pod: payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x 51m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 51m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-jjg48 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-23cde229-7646fc45dc Created pod: payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr 51m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 5 to 4 51m Normal Created pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Container created 51m Normal Started pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Container started 51m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Stopping container istio-proxy 51m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Startup probe error: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Startup probe failed: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Startup probe error: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Startup probe error: Get "http://10.132.0.91:15021/healthz/ready": dial tcp 10.132.0.91:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Startup probe failed: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Startup probe failed: Get "http://10.132.0.91:15021/healthz/ready": dial tcp 10.132.0.91:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Startup probe failed: Get "http://10.134.0.131:15021/healthz/ready": dial tcp 10.134.0.131:15021: connect: connection refused 51m Normal Created pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Container created 51m Normal Started pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Container started 51m Normal AddedInterface pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Add eth0 [10.134.0.134/23] from ovn-kubernetes 51m Normal Pulled pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Pulled pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Container created 51m Normal Started pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Container started 51m Normal AddedInterface pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Add eth0 [10.134.0.135/23] from ovn-kubernetes 51m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Startup probe error: Get "http://10.134.0.131:15021/healthz/ready": dial tcp 10.134.0.131:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Startup probe failed: Get "http://10.132.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Startup probe error: Get "http://10.132.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Readiness probe failed: HTTP probe failed with statuscode: 503 51m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-jjg48 Readiness probe error: HTTP probe failed with statuscode: 503... 51m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Startup probe error: Get "http://10.133.0.57:15021/healthz/ready": dial tcp 10.133.0.57:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Startup probe failed: Get "http://10.133.0.57:15021/healthz/ready": dial tcp 10.133.0.57:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Startup probe failed: Get "http://10.132.0.93:15021/healthz/ready": dial tcp 10.132.0.93:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Startup probe error: Get "http://10.132.0.93:15021/healthz/ready": dial tcp 10.132.0.93:15021: connect: connection refused... 51m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 51m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Startup probe error: Get "http://10.133.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 51m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-pzcsw 51m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 51m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Add eth0 [10.134.0.136/23] from ovn-kubernetes 51m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.644s (3.644s including waiting). Image size: 185476477 bytes. 51m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Container created 51m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Container started 51m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 51m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Readiness probe error: HTTP probe failed with statuscode: 503... 51m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Readiness probe failed: HTTP probe failed with statuscode: 503 51m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-ctppg 51m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-ctppg Stopping container istio-proxy 51m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 4 to 3 50m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-2qgzh 50m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 3 to 6 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Container created 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Container started 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Add eth0 [10.134.0.138/23] from ovn-kubernetes 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-695mh Add eth0 [10.133.0.60/23] from ovn-kubernetes 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-695mh Container started 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-695mh Container created 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Container started 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Container created 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Add eth0 [10.132.0.94/23] from ovn-kubernetes 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-695mh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Startup probe failed: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-695mh Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Startup probe error: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Startup probe failed: Get "http://10.134.0.138:15021/healthz/ready": dial tcp 10.134.0.138:15021: connect: connection refused 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Startup probe error: Get "http://10.134.0.138:15021/healthz/ready": dial tcp 10.134.0.138:15021: connect: connection refused... 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-695mh Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-695mh Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-695mh Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 (combined from similar events): Created pod: maas-default-gateway-openshift-default-6886b96f9-xjvz5 50m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default (combined from similar events): Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 6 to 10 50m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Add eth0 [10.132.0.96/23] from ovn-kubernetes 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Container started 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Container created 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Container started 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Add eth0 [10.133.0.62/23] from ovn-kubernetes 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Container created 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Container created 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Container started 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Add eth0 [10.132.0.97/23] from ovn-kubernetes 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Container started 50m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Container created 50m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Add eth0 [10.134.0.139/23] from ovn-kubernetes 50m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Startup probe failed: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Startup probe error: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Startup probe failed: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Created pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 50m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled up replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 6 to 9 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Startup probe error: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused... 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 9; reason: cpu resource utilization (percentage of request) above target 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 50m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Add eth0 [10.133.0.64/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Container started 50m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Container started 50m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Add eth0 [10.134.0.140/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Container created 50m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Add eth0 [10.132.0.98/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Container created 50m Normal Created pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Container created 50m Normal Started pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Container started 50m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Startup probe failed: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Startup probe error: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Startup probe failed: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Startup probe error: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Readiness probe failed: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Readiness probe error: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal Started pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Container started 50m Normal Created pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Container created 50m Normal Pulled pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning Unhealthy pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Readiness probe failed: Get "http://10.134.0.127:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Readiness probe error: Get "http://10.134.0.127:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal Created pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Container created 50m Normal Pulled pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Container created 50m Normal Started pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Container started 50m Normal Pulled pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Container started 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-23cde229-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-23cde229-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning ProbeError pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Startup probe error: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused... 49m Warning ProbeError pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Startup probe failed: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Startup probe failed: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused 49m Warning ProbeError pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Startup probe error: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused... 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-7f0ceebd-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-7f0ceebd-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-8660b337-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-8660b337-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-23cde229-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-23cde229-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-7f0ceebd-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-7f0ceebd-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-8660b337-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-8660b337-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Normal Killing pod/payload-processing-e2e-migrate-7f0ceebd-d6ff68448-hq2gz Stopping container payload-processing 49m Normal Killing pod/payload-pre-processing-e2e-migrate-7f0ceebd-57447cb9b7-lq6qg Stopping container payload-pre-processing 49m Normal Killing pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Stopping container payload-processing 49m Normal Killing pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Stopping container payload-pre-processing 49m Warning Unhealthy pod/payload-pre-processing-e2e-ait-8660b337-6b9886ff7f-vtn8p Readiness probe failed: timeout: failed to connect service "10.134.0.133:50052" within 1s: context deadline exceeded 49m Warning Unhealthy pod/payload-processing-e2e-ait-8660b337-f578c795b-z45vn Readiness probe failed: timeout: failed to connect service "10.132.0.92:50052" within 1s: context deadline exceeded 49m Normal Killing pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Stopping container payload-processing 49m Normal Killing pod/payload-pre-processing-e2e-ait-23cde229-7646fc45dc-nxsvr Stopping container payload-pre-processing 49m Warning Unhealthy pod/payload-processing-e2e-ait-23cde229-7498b68f66-tfn9x Readiness probe failed: timeout: failed to connect service "10.134.0.135:50052" within 1s: context deadline exceeded 48m Normal Killing pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Stopping container istio-proxy 48m Warning ProbeError pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Readiness probe error: HTTP probe failed with statuscode: 503... 48m Warning Unhealthy pod/e2e-migrate-7f0ceebd-gw-openshift-default-76c486c9df-z4zxh Readiness probe failed: HTTP probe failed with statuscode: 503 48m Normal ScalingReplicaSet deployment/e2e-ait-0d757729-openshift-default Scaled up replica set e2e-ait-0d757729-openshift-default-85cb94cb6c from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-ait-0d757729-openshift-default-85cb94cb6c Created pod: e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 48m Normal AddedLabel gateway/e2e-ait-0d757729 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-0d757729 48m Normal Killing pod/e2e-ait-8660b337-openshift-default-5b69bc746c-r68rg Stopping container istio-proxy 48m Normal AddedInterface pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Add eth0 [10.132.0.99/23] from ovn-kubernetes 48m Normal AddedLabel gateway/e2e-derive-c2bdb58a Added label istio.io/rev=openshift-gateway to gateway e2e-derive-c2bdb58a 48m Normal ScalingReplicaSet deployment/e2e-derive-c2bdb58a-openshift-default Scaled up replica set e2e-derive-c2bdb58a-openshift-default-75cd86dcc from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-derive-c2bdb58a-openshift-default-75cd86dcc Created pod: e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v 48m Normal AddedInterface pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Add eth0 [10.132.0.100/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-0d757729 Scaled up replica set payload-pre-processing-e2e-ait-0d757729-66698c964c from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-0d757729 Scaled up replica set payload-processing-e2e-ait-0d757729-7857666684 from 0 to 1 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-0d757729-66698c964c Error creating: pods "payload-pre-processing-e2e-ait-0d757729-66698c964c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-0d757729: serviceaccount "payload-processing-e2e-ait-0d757729" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-0d757729-7857666684 Created pod: payload-processing-e2e-ait-0d757729-7857666684-6pxgx 48m Normal Killing pod/e2e-ait-23cde229-openshift-default-659c875689-4fjrs Stopping container istio-proxy 48m Warning FailedCreate replicaset/payload-processing-e2e-ait-0d757729-7857666684 Error creating: pods "payload-processing-e2e-ait-0d757729-7857666684-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-0d757729: serviceaccount "payload-processing-e2e-ait-0d757729" not found 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-0d757729-66698c964c Created pod: payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 48m Normal Created pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Container created 48m Normal Started pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Container started 48m Normal Pulled pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal AddedInterface pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Add eth0 [10.132.0.102/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Container created 48m Normal Started pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Container started 48m Normal AddedInterface pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Add eth0 [10.132.0.101/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-c2bdb58a Scaled up replica set payload-processing-e2e-derive-c2bdb58a-65b4844974 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-c2bdb58a Scaled up replica set payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79 from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79 Created pod: payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-c2bdb58a-65b4844974 Created pod: payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79 Error creating: pods "payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-c2bdb58a: serviceaccount "payload-processing-e2e-derive-c2bdb58a" not found 48m Warning FailedCreate replicaset/payload-processing-e2e-derive-c2bdb58a-65b4844974 Error creating: pods "payload-processing-e2e-derive-c2bdb58a-65b4844974-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-c2bdb58a: serviceaccount "payload-processing-e2e-derive-c2bdb58a" not found 48m Normal Pulling pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 48m Normal AddedInterface pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Add eth0 [10.133.0.65/23] from ovn-kubernetes 48m Normal Pulling pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 48m Normal AddedInterface pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Add eth0 [10.133.0.66/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.262s (3.262s including waiting). Image size: 125353965 bytes. 48m Normal Pulled pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.202s (3.202s including waiting). Image size: 125353965 bytes. 48m Normal Started pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Container started 48m Normal Created pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Container created 48m Normal Created pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Container created 48m Normal Started pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Container started 47m Normal ScalingReplicaSet deployment/e2e-mt-c650f2b7-openshift-default Scaled up replica set e2e-mt-c650f2b7-openshift-default-59ccc6c545 from 0 to 1 47m Normal SuccessfulCreate replicaset/e2e-mt-c650f2b7-openshift-default-59ccc6c545 Created pod: e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk 47m Normal AddedLabel gateway/e2e-mt-c650f2b7 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-c650f2b7 47m Normal AddedInterface pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Add eth0 [10.134.0.152/23] from ovn-kubernetes 47m Normal AddedLabel gateway/e2e-shared-a-w1-d71c23 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-d71c23 47m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c Created pod: e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p 47m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-d71c23-openshift-default Scaled up replica set e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c from 0 to 1 47m Normal AddedInterface pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Add eth0 [10.134.0.153/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-c650f2b7 Scaled up replica set payload-processing-e2e-mt-c650f2b7-7fc9f56798 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-c650f2b7 Scaled up replica set payload-pre-processing-e2e-mt-c650f2b7-75749c78cf from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-mt-c650f2b7-7fc9f56798 Error creating: pods "payload-processing-e2e-mt-c650f2b7-7fc9f56798-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c650f2b7: serviceaccount "payload-processing-e2e-mt-c650f2b7" not found 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf Created pod: payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-c650f2b7-7fc9f56798 Created pod: payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf Error creating: pods "payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c650f2b7: serviceaccount "payload-processing-e2e-mt-c650f2b7" not found 47m Normal Pulled pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Container started 47m Normal AddedInterface pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Add eth0 [10.132.0.105/23] from ovn-kubernetes 47m Normal Created pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Container created 47m Normal Pulled pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Container started 47m Normal AddedInterface pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Add eth0 [10.133.0.69/23] from ovn-kubernetes 47m Normal Created pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Container created 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-d71c23 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-d71c23 Scaled up replica set payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5 Created pod: payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d Error creating: pods "payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23: serviceaccount "payload-processing-e2e-shared-a-w1-d71c23" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d Created pod: payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23: serviceaccount "payload-processing-e2e-shared-a-w1-d71c23" not found 47m Normal Pulled pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Container started 47m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Container created 47m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Add eth0 [10.132.0.106/23] from ovn-kubernetes 47m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Add eth0 [10.133.0.70/23] from ovn-kubernetes 47m Normal Started pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Container started 47m Normal Created pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Container created 47m Normal Started pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Container started 47m Normal Created pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Container created 47m Normal Pulled pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-d71c23-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-d71c23-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning Unhealthy pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Startup probe failed: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused 47m Warning ProbeError pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused... 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-c2bdb58a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-c2bdb58a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning Unhealthy pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Readiness probe failed: Get "http://10.132.0.99:15021/healthz/ready": read tcp 10.132.0.2:34334->10.132.0.99:15021: read: connection reset by peer 47m Warning ProbeError pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Readiness probe error: Get "http://10.132.0.99:15021/healthz/ready": read tcp 10.132.0.2:34334->10.132.0.99:15021: read: connection reset by peer... 47m Normal Pulled pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Started pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Container started 47m Normal Created pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Container created 47m Normal AddedLabel gateway/e2e-mt-27219c73 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-27219c73 47m Normal SuccessfulCreate replicaset/e2e-mt-27219c73-openshift-default-88d599849 Created pod: e2e-mt-27219c73-openshift-default-88d599849-rnbxz 47m Normal ScalingReplicaSet deployment/e2e-mt-27219c73-openshift-default Scaled up replica set e2e-mt-27219c73-openshift-default-88d599849 from 0 to 1 47m Normal AddedInterface pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Add eth0 [10.134.0.156/23] from ovn-kubernetes 47m Normal Killing pod/payload-pre-processing-e2e-ait-0d757729-66698c964c-z9ww7 Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-ait-0d757729-7857666684-6pxgx Stopping container payload-processing 47m Warning ProbeError pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Startup probe error: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Startup probe failed: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused 46m Normal Killing pod/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf Stopping container payload-processing 46m Normal Killing pod/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h Stopping container payload-pre-processing 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-c2bdb58a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-c2bdb58a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal Killing pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m Stopping container istio-proxy 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-27219c73 Scaled up replica set payload-pre-processing-e2e-mt-27219c73-664dc987f5 from 0 to 1 46m Normal ScalingReplicaSet deployment/e2e-worker-w5-a78bf4-openshift-default Scaled down replica set e2e-worker-w5-a78bf4-openshift-default-79b556ccdc from 3 to 2 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default New size: 2; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc Deleted pod: e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m 46m Warning FailedCreate replicaset/payload-processing-e2e-mt-27219c73-57f8548544 Error creating: pods "payload-processing-e2e-mt-27219c73-57f8548544-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-27219c73: serviceaccount "payload-processing-e2e-mt-27219c73" not found 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-27219c73 Scaled up replica set payload-processing-e2e-mt-27219c73-57f8548544 from 0 to 1 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-27219c73-664dc987f5 Error creating: pods "payload-pre-processing-e2e-mt-27219c73-664dc987f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-27219c73: serviceaccount "payload-processing-e2e-mt-27219c73" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-27219c73-664dc987f5 Created pod: payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-27219c73-57f8548544 Created pod: payload-processing-e2e-mt-27219c73-57f8548544-mxwzh 46m Normal AddedInterface pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Add eth0 [10.132.0.108/23] from ovn-kubernetes 46m Normal Created pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Container created 46m Normal Pulled pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Add eth0 [10.132.0.107/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Container started 46m Normal Pulled pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Started pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Container started 46m Normal Created pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Container created 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-0d757729-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-0d757729-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal Killing pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Stopping container istio-proxy 46m Normal Killing pod/e2e-derive-c2bdb58a-openshift-default-75cd86dcc-qcr6v Stopping container istio-proxy 46m Normal AddedLabel gateway/e2e-shared-a-w2-99c030 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-99c030 46m Normal AddedLabel gateway/e2e-shared-a-w5-0ac4e5 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-0ac4e5 46m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-0ac4e5-openshift-default Scaled up replica set e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5 Created pod: e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 46m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-99c030-openshift-default Scaled up replica set e2e-shared-a-w2-99c030-openshift-default-7949dd7947 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-99c030-openshift-default-7949dd7947 Created pod: e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm 46m Normal AddedInterface pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Add eth0 [10.132.0.109/23] from ovn-kubernetes 46m Normal AddedInterface pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Add eth0 [10.133.0.73/23] from ovn-kubernetes 46m Warning Unhealthy pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Readiness probe failed: HTTP probe failed with statuscode: 503 46m Warning ProbeError pod/e2e-ait-0d757729-openshift-default-85cb94cb6c-nkg89 Readiness probe error: HTTP probe failed with statuscode: 503... 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-0ac4e5 Scaled up replica set payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9 from 0 to 1 46m Normal Killing pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j Stopping container istio-proxy 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-0ac4e5 Scaled up replica set payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d from 0 to 1 46m Normal ScalingReplicaSet deployment/e2e-worker-w2-df031b-openshift-default Scaled down replica set e2e-worker-w2-df031b-openshift-default-77cd44ff85 from 3 to 2 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default New size: 2; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-df031b-openshift-default-77cd44ff85 Deleted pod: e2e-worker-w2-df031b-openshift-default-77cd44ff85-txq2j 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9 Created pod: payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9 Error creating: pods "payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5: serviceaccount "payload-processing-e2e-shared-a-w5-0ac4e5" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d Created pod: payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d Error creating: pods "payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-0ac4e5: serviceaccount "payload-processing-e2e-shared-a-w5-0ac4e5" not found 46m Normal Created pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Container created 46m Normal Started pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Container started 46m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Container created 46m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Add eth0 [10.132.0.110/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Container started 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Add eth0 [10.132.0.111/23] from ovn-kubernetes 46m Normal Pulled pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Killing pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g Stopping container istio-proxy 46m Normal ScalingReplicaSet deployment/e2e-worker-w4-8e2fc1-openshift-default Scaled down replica set e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f from 3 to 2 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default New size: 2; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f Deleted pod: e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-sxm8g 46m Normal Started pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Container started 46m Normal Pulled pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Container created 46m Normal Pulled pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Container started 46m Normal Created pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Container created 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7 Error creating: pods "payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-99c030: serviceaccount "payload-processing-e2e-shared-a-w2-99c030" not found 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-99c030 Scaled up replica set payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76 from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-99c030 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7 Created pod: payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76 Error creating: pods "payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-99c030: serviceaccount "payload-processing-e2e-shared-a-w2-99c030" not found 45m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Add eth0 [10.132.0.114/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76 Created pod: payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Add eth0 [10.132.0.113/23] from ovn-kubernetes 45m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Container created 45m Normal Pulled pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Container created 45m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-c650f2b7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Started pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Container started 45m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Container started 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-c650f2b7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning ProbeError pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Startup probe error: Get "http://10.134.0.153:15021/healthz/ready": dial tcp 10.134.0.153:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Startup probe error: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Startup probe failed: Get "http://10.134.0.153:15021/healthz/ready": dial tcp 10.134.0.153:15021: connect: connection refused 45m Warning Unhealthy pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Startup probe failed: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 6; reason: All metrics below target 45m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled down replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 10 to 6 45m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-mwm9n 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-hhmdt 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-rvh6r 45m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh 45m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled down replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 10 to 6 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 6; reason: All metrics below target 45m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs Stopping container istio-proxy 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default New size: 4; reason: All metrics below target 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Deleted pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-6q5xs 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Deleted pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled down replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 6 to 4 45m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-429wn Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-hs7bf Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb Stopping container istio-proxy 45m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-n4wrr Readiness probe error: HTTP probe failed with statuscode: 503... 45m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Readiness probe error: HTTP probe failed with statuscode: 503... 45m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-292xh Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Readiness probe error: HTTP probe failed with statuscode: 503... 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-0ac4e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qlxnd Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-0ac4e5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-d71c23-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-d71c23-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 45m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 45m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-zd4sq 45m Normal Created pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Container created 45m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Add eth0 [10.132.0.115/23] from ovn-kubernetes 45m Normal Started pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Container started 45m Normal Pulled pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 45m Normal AddedLabel gateway/e2e-shared-b-w1-061bda Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-061bda 45m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.2s (4.2s including waiting). Image size: 185476477 bytes. 45m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Container created 45m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Container started 45m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-061bda-openshift-default Scaled up replica set e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9 from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9 Created pod: e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t 45m Warning Unhealthy pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Startup probe failed: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused 45m Warning ProbeError pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Startup probe error: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused... 45m Normal AddedInterface pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Add eth0 [10.134.0.157/23] from ovn-kubernetes 45m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Readiness probe error: HTTP probe failed with statuscode: 503... 45m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-c650f2b7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-c650f2b7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal AddedLabel gateway/e2e-shared-a-w3-034eba Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-034eba 45m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-034eba-openshift-default Scaled up replica set e2e-shared-a-w3-034eba-openshift-default-67449b85b7 from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-034eba-openshift-default-67449b85b7 Created pod: e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s 45m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-061bda-758d759764 Error creating: pods "payload-processing-e2e-shared-b-w1-061bda-758d759764-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-061bda: serviceaccount "payload-processing-e2e-shared-b-w1-061bda" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-061bda-758d759764 Created pod: payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-061bda Scaled up replica set payload-processing-e2e-shared-b-w1-061bda-758d759764 from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-061bda Scaled up replica set payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f from 0 to 1 45m Normal AddedInterface pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Add eth0 [10.134.0.158/23] from ovn-kubernetes 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f Error creating: pods "payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-061bda: serviceaccount "payload-processing-e2e-shared-b-w1-061bda" not found 45m Normal Created pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Container created 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Add eth0 [10.133.0.74/23] from ovn-kubernetes 45m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Add eth0 [10.132.0.116/23] from ovn-kubernetes 45m Normal Pulled pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Container started 45m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Container created 45m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Container started 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f Created pod: payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-034eba Scaled up replica set payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5 from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb Created pod: payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-034eba Scaled up replica set payload-processing-e2e-shared-a-w3-034eba-6b75945bcb from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb Error creating: pods "payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-034eba: serviceaccount "payload-processing-e2e-shared-a-w3-034eba" not found 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-034eba: serviceaccount "payload-processing-e2e-shared-a-w3-034eba" not found 45m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Add eth0 [10.132.0.117/23] from ovn-kubernetes 45m Normal Pulled pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Container created 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5 Created pod: payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n 45m Normal Started pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Container started 45m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Container created 45m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Container started 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Add eth0 [10.132.0.118/23] from ovn-kubernetes 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-27219c73-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-27219c73-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv 45m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff 45m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-q8h4s 45m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-8x77b 45m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-v6rff Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-mhxp6 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 4; reason: All metrics below target 45m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled down replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 9 to 4 44m Warning ProbeError pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Readiness probe error: Get "http://10.132.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Readiness probe failed: Get "http://10.132.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Warning ProbeError pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Readiness probe error: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Readiness probe failed: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal Pulled pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Pulled pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Container started 44m Normal Created pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Container created 44m Normal Started pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Container started 44m Normal Created pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Container created 44m Warning ProbeError pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Startup probe failed: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused 44m Warning ProbeError pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Startup probe error: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused... 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-99c030-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-99c030-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-99c030-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-processing-e2e-mt-c650f2b7-7fc9f56798-9l7pg Stopping container payload-processing 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-0ac4e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-99c030-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-0ac4e5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal Killing pod/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l Stopping container payload-pre-processing 43m Normal AddedLabel gateway/e2e-shared-b-w5-ab9c38 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-ab9c38 43m Normal Started pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Container started 43m Normal Created pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Container created 43m Normal Pulled pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5 Created pod: e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf 43m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-ab9c38-openshift-default Scaled up replica set e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5 from 0 to 1 43m Normal AddedInterface pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Add eth0 [10.134.0.161/23] from ovn-kubernetes 43m Normal AddedLabel gateway/e2e-shared-b-w2-f8ac60 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-f8ac60 43m Warning Unhealthy pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Startup probe failed: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused 43m Warning ProbeError pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Startup probe error: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused... 43m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497 Created pod: e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h 43m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-f8ac60-openshift-default Scaled up replica set e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497 from 0 to 1 43m Normal AddedInterface pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Add eth0 [10.134.0.162/23] from ovn-kubernetes 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-ab9c38 Scaled up replica set payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-ab9c38 Scaled up replica set payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc from 0 to 1 43m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b Error creating: pods "payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38: serviceaccount "payload-processing-e2e-shared-b-w5-ab9c38" not found 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc Error creating: pods "payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-ab9c38: serviceaccount "payload-processing-e2e-shared-b-w5-ab9c38" not found 43m Normal Created pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Container created 43m Normal Pulled pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Add eth0 [10.132.0.122/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Container started 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b Created pod: payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp 43m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Add eth0 [10.133.0.77/23] from ovn-kubernetes 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc Created pod: payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 43m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Container created 43m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Container started 43m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-f8ac60 Scaled up replica set payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-f8ac60 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959 from 0 to 1 43m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d Error creating: pods "payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60: serviceaccount "payload-processing-e2e-shared-b-w2-f8ac60" not found 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959 Error creating: pods "payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60: serviceaccount "payload-processing-e2e-shared-b-w2-f8ac60" not found 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959 Created pod: payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d Created pod: payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx 43m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Add eth0 [10.133.0.78/23] from ovn-kubernetes 43m Normal Created pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Container created 43m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Add eth0 [10.133.0.79/23] from ovn-kubernetes 43m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Started pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Container started 43m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Container started 43m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Container created 43m Normal Pulled pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-061bda-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-061bda-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal Killing pod/e2e-mt-c650f2b7-openshift-default-59ccc6c545-ss2rk Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-api-a-c3d13a-openshift-default Scaled up replica set e2e-api-a-c3d13a-openshift-default-7b6b44f5f from 0 to 1 43m Normal AddedLabel gateway/e2e-api-a-c3d13a Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-c3d13a 43m Normal SuccessfulCreate replicaset/e2e-api-a-c3d13a-openshift-default-7b6b44f5f Created pod: e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j 43m Normal AddedInterface pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Add eth0 [10.134.0.163/23] from ovn-kubernetes 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-061bda-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-061bda-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Started pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Container started 43m Normal Pulled pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Created pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Container created 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-c3d13a Scaled up replica set payload-pre-processing-e2e-api-a-c3d13a-799f746d9f from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-c3d13a Scaled up replica set payload-processing-e2e-api-a-c3d13a-d7f5b8d76 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-c3d13a-d7f5b8d76 Created pod: payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc 43m Warning FailedCreate replicaset/payload-processing-e2e-api-a-c3d13a-d7f5b8d76 Error creating: pods "payload-processing-e2e-api-a-c3d13a-d7f5b8d76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-c3d13a: serviceaccount "payload-processing-e2e-api-a-c3d13a" not found 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f Created pod: payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f Error creating: pods "payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-c3d13a: serviceaccount "payload-processing-e2e-api-a-c3d13a" not found 43m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Add eth0 [10.133.0.80/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Container started 43m Normal Created pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Container created 43m Normal Pulled pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Add eth0 [10.133.0.81/23] from ovn-kubernetes 43m Normal Started pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Container started 43m Normal Created pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Container created 43m Normal Pulled pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Startup probe error: Get "http://10.134.0.158:15021/healthz/ready": dial tcp 10.134.0.158:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Startup probe failed: Get "http://10.134.0.158:15021/healthz/ready": dial tcp 10.134.0.158:15021: connect: connection refused 42m Normal ScalingReplicaSet deployment/e2e-inf-a-3f09ec-openshift-default Scaled up replica set e2e-inf-a-3f09ec-openshift-default-85c5dcc74c from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c Created pod: e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm 42m Normal AddedLabel gateway/e2e-inf-a-3f09ec Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-3f09ec 42m Normal AddedInterface pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Add eth0 [10.134.0.165/23] from ovn-kubernetes 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-034eba-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-034eba-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-3f09ec Scaled up replica set payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579 from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579 Error creating: pods "payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-3f09ec: serviceaccount "payload-processing-e2e-inf-a-3f09ec" not found 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-3f09ec-8677556bb4 Error creating: pods "payload-processing-e2e-inf-a-3f09ec-8677556bb4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-3f09ec: serviceaccount "payload-processing-e2e-inf-a-3f09ec" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-3f09ec-8677556bb4 Created pod: payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-3f09ec Scaled up replica set payload-processing-e2e-inf-a-3f09ec-8677556bb4 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579 Created pod: payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh 42m Normal Started pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Container started 42m Normal Created pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Container created 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Add eth0 [10.132.0.123/23] from ovn-kubernetes 42m Normal AddedLabel gateway/e2e-shared-b-w3-712099 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-712099 42m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-712099-openshift-default Scaled up replica set e2e-shared-b-w3-712099-openshift-default-7c59b44487 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-712099-openshift-default-7c59b44487 Created pod: e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml 42m Normal Started pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Container started 42m Normal Created pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Container created 42m Normal Pulled pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Add eth0 [10.133.0.82/23] from ovn-kubernetes 42m Normal AddedInterface pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Add eth0 [10.134.0.166/23] from ovn-kubernetes 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-034eba-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-034eba-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-712099-7949d459d4 Error creating: pods "payload-processing-e2e-shared-b-w3-712099-7949d459d4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-712099: serviceaccount "payload-processing-e2e-shared-b-w3-712099" not found 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-712099 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7 from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-712099 Scaled up replica set payload-processing-e2e-shared-b-w3-712099-7949d459d4 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-712099-7949d459d4 Created pod: payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7 Error creating: pods "payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-712099: serviceaccount "payload-processing-e2e-shared-b-w3-712099" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7 Created pod: payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz 42m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Container created 42m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Add eth0 [10.133.0.83/23] from ovn-kubernetes 42m Normal Pulled pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Container created 42m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Add eth0 [10.133.0.84/23] from ovn-kubernetes 42m Normal Started pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Container started 42m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Container started 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-3f09ec-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-3f09ec-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Readiness probe error: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Readiness probe error: Get "http://10.132.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Readiness probe failed: Get "http://10.132.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Readiness probe failed: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Normal Killing pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Stopping container payload-processing 41m Normal Killing pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Stopping container payload-pre-processing 41m Normal Created pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Container created 41m Normal Pulled pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Warning Unhealthy pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Readiness probe failed: Get "http://10.134.0.161:15021/healthz/ready": read tcp 10.134.0.2:50510->10.134.0.161:15021: read: connection reset by peer 41m Warning ProbeError pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Readiness probe error: Get "http://10.134.0.161:15021/healthz/ready": read tcp 10.134.0.2:50510->10.134.0.161:15021: read: connection reset by peer... 41m Normal Started pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Container started 41m Normal Started pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Container started 41m Normal Pulled pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Container created 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-f8ac60-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-f8ac60-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning ProbeError pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Startup probe error: Get "http://10.134.0.162:15021/healthz/ready": dial tcp 10.134.0.162:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Startup probe failed: Get "http://10.134.0.162:15021/healthz/ready": dial tcp 10.134.0.162:15021: connect: connection refused 41m Warning ProbeError pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Startup probe error: Get "http://10.134.0.161:15021/healthz/ready": dial tcp 10.134.0.161:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Startup probe failed: Get "http://10.134.0.161:15021/healthz/ready": dial tcp 10.134.0.161:15021: connect: connection refused 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-f8ac60-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-f8ac60-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning Unhealthy pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-s4bdb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Warning Unhealthy pod/payload-processing-e2e-mt-27219c73-57f8548544-mxwzh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Normal AddedInterface pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Add eth0 [10.134.0.168/23] from ovn-kubernetes 41m Normal ScalingReplicaSet deployment/e2e-api-a-3f0723-openshift-default Scaled up replica set e2e-api-a-3f0723-openshift-default-898ff695 from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-api-a-3f0723-openshift-default-898ff695 Created pod: e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 41m Normal AddedLabel gateway/e2e-api-a-3f0723 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-3f0723 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-3f0723-5bf4b7db56 Created pod: payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-3f0723 Scaled up replica set payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c Error creating: pods "payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3f0723: serviceaccount "payload-processing-e2e-api-a-3f0723" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c Created pod: payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm 41m Warning FailedCreate replicaset/payload-processing-e2e-api-a-3f0723-5bf4b7db56 Error creating: pods "payload-processing-e2e-api-a-3f0723-5bf4b7db56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3f0723: serviceaccount "payload-processing-e2e-api-a-3f0723" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-3f0723 Scaled up replica set payload-processing-e2e-api-a-3f0723-5bf4b7db56 from 0 to 1 41m Normal Pulled pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Add eth0 [10.132.0.127/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Add eth0 [10.132.0.126/23] from ovn-kubernetes 41m Normal Created pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Container created 41m Normal Started pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Container started 41m Normal Created pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Container created 41m Normal Started pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Container started 41m Normal Started pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Container started 41m Normal Created pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Container created 41m Normal Pulled pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Warning ProbeError pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Startup probe error: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Startup probe failed: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-ab9c38-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-ab9c38-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-c3d13a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-c3d13a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 3; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-df031b-openshift-default-77cd44ff85 Deleted pod: e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl 40m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-czdrl Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-df031b-openshift-default New size: 1; reason: All metrics below target 40m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled down replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 6 to 3 40m Normal ScalingReplicaSet deployment/e2e-worker-w2-df031b-openshift-default Scaled down replica set e2e-worker-w2-df031b-openshift-default-77cd44ff85 from 2 to 1 40m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled down replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 6 to 3 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a78bf4-openshift-default New size: 1; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc Deleted pod: e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv 40m Normal Killing pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-252lv Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w4-8e2fc1-openshift-default Scaled down replica set e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f from 2 to 1 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-8e2fc1-openshift-default New size: 1; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f Deleted pod: e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 40m Normal ScalingReplicaSet deployment/e2e-worker-w1-5f9b95-openshift-default Scaled down replica set e2e-worker-w1-5f9b95-openshift-default-865b85b75 from 4 to 2 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5f9b95-openshift-default New size: 2; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Stopping container istio-proxy 40m Normal Started pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Container started 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 3; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Deleted pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c 40m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Readiness probe error: HTTP probe failed with statuscode: 503... 40m Normal ScalingReplicaSet deployment/e2e-worker-w5-a78bf4-openshift-default Scaled down replica set e2e-worker-w5-a78bf4-openshift-default-79b556ccdc from 2 to 1 40m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-7k7nt Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Stopping container istio-proxy 40m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-5f9b95-openshift-default-865b85b75 Deleted pod: e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 40m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qj7h7 Stopping container istio-proxy 40m Normal Pulled pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Container created 40m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning Unhealthy pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning ProbeError pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-6tsqj Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning ProbeError pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning Unhealthy pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Startup probe failed: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Startup probe error: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused... 40m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled down replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 4 to 3 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 3; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng 40m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng Stopping container istio-proxy 40m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-0ac4e5-6c5dc88dc9-wdjhp Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-shared-a-w5-0ac4e5-69c854957d-65d66 Stopping container payload-processing 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-c3d13a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-c3d13a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal Killing pod/payload-processing-e2e-shared-a-w2-99c030-7c4b5c4f76-fr4dl Stopping container payload-processing 40m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-99c030-7894dcd9b7-sq8s4 Stopping container payload-pre-processing 40m Normal Created pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Container created 40m Normal Pulled pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Started pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Container started 40m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-zd4sq Stopping container discovery 40m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-zd4sq 40m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 40m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 40m Warning Unhealthy pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Startup probe failed: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused 40m Warning ProbeError pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Startup probe error: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused... 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-712099-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-712099-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-3f09ec-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-3f09ec-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled down replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 3 to 2 40m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 2; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Stopping container istio-proxy 40m Normal Killing pod/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc Stopping container payload-processing 40m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-c5ljp Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-712099-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-712099-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Killing pod/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh Stopping container payload-pre-processing 39m Normal Killing pod/payload-processing-e2e-inf-a-3f09ec-8677556bb4-zhzws Stopping container payload-processing 39m Normal Killing pod/e2e-shared-a-w5-0ac4e5-openshift-default-6b5f7c54b5-2n5n9 Stopping container istio-proxy 39m Normal Killing pod/payload-processing-e2e-shared-a-w3-034eba-6b75945bcb-76pn4 Stopping container payload-processing 39m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-034eba-785f5645c5-rxk5n Stopping container payload-pre-processing 39m Normal Killing pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm Stopping container istio-proxy 39m Normal Killing pod/payload-processing-e2e-shared-b-w5-ab9c38-5d666cd85b-xtrfp Stopping container payload-processing 39m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6 Stopping container payload-pre-processing 39m Normal Killing pod/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx Stopping container payload-processing 39m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2 Stopping container payload-pre-processing 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-27219c73 Scaled up replica set payload-processing-e2e-mt-27219c73-57f8548544 from 0 to 1 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-27219c73-664dc987f5 Error creating: pods "payload-pre-processing-e2e-mt-27219c73-664dc987f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-27219c73: serviceaccount "payload-processing-e2e-mt-27219c73" not found 39m Normal Pulled pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-27219c73 Scaled up replica set payload-pre-processing-e2e-mt-27219c73-664dc987f5 from 0 to 1 39m Normal Created pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Container created 39m Normal Started pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Container started 39m Warning FailedCreate replicaset/payload-processing-e2e-mt-27219c73-57f8548544 Error creating: pods "payload-processing-e2e-mt-27219c73-57f8548544-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-27219c73: serviceaccount "payload-processing-e2e-mt-27219c73" not found 39m Normal AddedInterface pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Add eth0 [10.132.0.129/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-27219c73-57f8548544 Created pod: payload-processing-e2e-mt-27219c73-57f8548544-drklv 39m Normal Created pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Container created 39m Normal Pulled pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Container started 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-27219c73-664dc987f5 Created pod: payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 39m Normal Created pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Container created 39m Normal AddedInterface pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Add eth0 [10.132.0.130/23] from ovn-kubernetes 39m Normal Pulled pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Container started 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-3f0723-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-3f0723-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning Unhealthy pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Startup probe failed: Get "http://10.134.0.168:15021/healthz/ready": dial tcp 10.134.0.168:15021: connect: connection refused 39m Warning ProbeError pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Startup probe error: Get "http://10.134.0.168:15021/healthz/ready": dial tcp 10.134.0.168:15021: connect: connection refused... 38m Normal Killing pod/e2e-api-a-c3d13a-openshift-default-7b6b44f5f-b7p7j Stopping container istio-proxy 38m Normal ScalingReplicaSet deployment/e2e-inf-a-bcef0e-openshift-default Scaled up replica set e2e-inf-a-bcef0e-openshift-default-58c977d9fc from 0 to 1 38m Normal SuccessfulCreate replicaset/e2e-inf-a-bcef0e-openshift-default-58c977d9fc Created pod: e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv 38m Normal AddedLabel gateway/e2e-inf-a-bcef0e Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-bcef0e 38m Normal AddedInterface pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Add eth0 [10.132.0.131/23] from ovn-kubernetes 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-3f0723-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-3f0723-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-bcef0e Scaled up replica set payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679 from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-bcef0e Scaled up replica set payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6 from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679 Created pod: payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Add eth0 [10.132.0.132/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6 Created pod: payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6 Error creating: pods "payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-bcef0e: serviceaccount "payload-processing-e2e-inf-a-bcef0e" not found 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679 Error creating: pods "payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-bcef0e: serviceaccount "payload-processing-e2e-inf-a-bcef0e" not found 38m Normal Created pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Container created 38m Normal Started pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Container started 38m Normal Pulled pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Started pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Container started 38m Normal Created pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Add eth0 [10.132.0.133/23] from ovn-kubernetes 38m Normal Killing pod/e2e-shared-a-w3-034eba-openshift-default-67449b85b7-xmk6s Stopping container istio-proxy 38m Normal AddedLabel gateway/e2e-api-b-65948c Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-65948c 38m Normal ScalingReplicaSet deployment/e2e-api-b-65948c-openshift-default Scaled up replica set e2e-api-b-65948c-openshift-default-b44f97585 from 0 to 1 38m Normal SuccessfulCreate replicaset/e2e-api-b-65948c-openshift-default-b44f97585 Created pod: e2e-api-b-65948c-openshift-default-b44f97585-gz54k 38m Normal Killing pod/e2e-inf-a-3f09ec-openshift-default-85c5dcc74c-2hxpm Stopping container istio-proxy 38m Normal AddedInterface pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Add eth0 [10.132.0.134/23] from ovn-kubernetes 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-65948c Scaled up replica set payload-pre-processing-e2e-api-b-65948c-7f8c699d95 from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-65948c Scaled up replica set payload-processing-e2e-api-b-65948c-79bcdcd958 from 0 to 1 38m Warning FailedCreate replicaset/payload-processing-e2e-api-b-65948c-79bcdcd958 Error creating: pods "payload-processing-e2e-api-b-65948c-79bcdcd958-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-65948c: serviceaccount "payload-processing-e2e-api-b-65948c" not found 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-65948c-7f8c699d95 Error creating: pods "payload-pre-processing-e2e-api-b-65948c-7f8c699d95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-65948c: serviceaccount "payload-processing-e2e-api-b-65948c" not found 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-65948c-7f8c699d95 Created pod: payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-65948c-79bcdcd958 Created pod: payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw 38m Normal AddedInterface pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Add eth0 [10.132.0.135/23] from ovn-kubernetes 38m Normal Killing pod/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4 Stopping container payload-processing 38m Normal Started pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Container started 38m Normal Created pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Container created 38m Normal Pulled pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz Stopping container payload-pre-processing 38m Normal Started pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Container started 38m Normal Pulled pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Container created 38m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Add eth0 [10.132.0.136/23] from ovn-kubernetes 38m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-d71c23-6f5dfc66f5-zh6jq Stopping container payload-pre-processing 38m Normal Killing pod/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng Stopping container payload-processing 38m Normal Killing pod/e2e-shared-b-w5-ab9c38-openshift-default-b8bc6f9f5-t5qkf Stopping container istio-proxy 37m Normal Killing pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Readiness probe failed: HTTP probe failed with statuscode: 503 37m Normal AddedLabel gateway/e2e-api-a-5dc13e Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-5dc13e 37m Warning ProbeError pod/e2e-shared-b-w2-f8ac60-openshift-default-5b8b9c497-kzw9h Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal ScalingReplicaSet deployment/e2e-api-a-5dc13e-openshift-default Scaled up replica set e2e-api-a-5dc13e-openshift-default-69d59fb4dc from 0 to 1 37m Normal SuccessfulCreate replicaset/e2e-api-a-5dc13e-openshift-default-69d59fb4dc Created pod: e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb 37m Normal AddedInterface pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Add eth0 [10.133.0.89/23] from ovn-kubernetes 37m Warning FailedCreate replicaset/payload-processing-e2e-api-a-5dc13e-84fc7d8688 Error creating: pods "payload-processing-e2e-api-a-5dc13e-84fc7d8688-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-5dc13e: serviceaccount "payload-processing-e2e-api-a-5dc13e" not found 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-5dc13e Scaled up replica set payload-processing-e2e-api-a-5dc13e-84fc7d8688 from 0 to 1 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-5dc13e-84fc7d8688 Created pod: payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5 Error creating: pods "payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-5dc13e: serviceaccount "payload-processing-e2e-api-a-5dc13e" not found 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-5dc13e Scaled up replica set payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5 from 0 to 1 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5 Created pod: payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf 37m Normal Pulled pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Add eth0 [10.133.0.91/23] from ovn-kubernetes 37m Normal Started pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Container started 37m Normal Created pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Container created 37m Normal Pulled pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Add eth0 [10.133.0.90/23] from ovn-kubernetes 37m Normal Started pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Container started 37m Normal Created pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Container created 37m Normal Killing pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-shared-a-w1-d71c23-openshift-default-669cb6f99c-t8n5p Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt Stopping container payload-pre-processing 37m Normal Killing pod/payload-processing-e2e-shared-b-w1-061bda-758d759764-9gjkh Stopping container payload-processing 37m Normal Killing pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Stopping container istio-proxy 37m Normal SuccessfulCreate replicaset/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5 Created pod: e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 37m Normal ScalingReplicaSet deployment/e2e-api-a-cd2d96-openshift-default Scaled up replica set e2e-api-a-cd2d96-openshift-default-5bb87dfdb5 from 0 to 1 37m Normal AddedLabel gateway/e2e-api-a-cd2d96 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-cd2d96 37m Warning Unhealthy pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Readiness probe failed: HTTP probe failed with statuscode: 503 37m Normal AddedInterface pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Add eth0 [10.132.0.138/23] from ovn-kubernetes 37m Warning ProbeError pod/e2e-shared-b-w3-712099-openshift-default-7c59b44487-hmlml Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Started pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Container started 36m Normal Created pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Container created 36m Normal Pulled pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 10 to 7 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-2qgzh 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-5bb9c 36m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-xjvz5 36m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Stopping container istio-proxy 36m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Stopping container istio-proxy 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-xjvz5 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-2qgzh Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-5bb9c Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-bcef0e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-bcef0e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-cd2d96 Scaled up replica set payload-processing-e2e-api-a-cd2d96-867bfcdccd from 0 to 1 36m Warning Unhealthy pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Startup probe failed: Get "http://10.132.0.131:15021/healthz/ready": dial tcp 10.132.0.131:15021: connect: connection refused 36m Warning ProbeError pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Startup probe error: Get "http://10.132.0.131:15021/healthz/ready": dial tcp 10.132.0.131:15021: connect: connection refused... 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-cd2d96-8c477746c Created pod: payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg 36m Warning FailedCreate replicaset/payload-processing-e2e-api-a-cd2d96-867bfcdccd Error creating: pods "payload-processing-e2e-api-a-cd2d96-867bfcdccd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-cd2d96: serviceaccount "payload-processing-e2e-api-a-cd2d96" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-cd2d96 Scaled up replica set payload-pre-processing-e2e-api-a-cd2d96-8c477746c from 0 to 1 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-cd2d96-8c477746c Error creating: pods "payload-pre-processing-e2e-api-a-cd2d96-8c477746c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-cd2d96: serviceaccount "payload-processing-e2e-api-a-cd2d96" not found 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-cd2d96-867bfcdccd Created pod: payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg 36m Normal Started pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Container started 36m Normal Started pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Container started 36m Normal Created pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Container created 36m Normal Created pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Container created 36m Normal Pulled pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Pulled pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Add eth0 [10.133.0.94/23] from ovn-kubernetes 36m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Add eth0 [10.132.0.139/23] from ovn-kubernetes 36m Normal Killing pod/payload-pre-processing-e2e-mt-27219c73-664dc987f5-gvwm7 Stopping container payload-pre-processing 36m Normal Killing pod/payload-processing-e2e-mt-27219c73-57f8548544-drklv Stopping container payload-processing 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-bcef0e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-bcef0e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal Pulled pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Container started 36m Normal Created pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Container created 36m Warning Unhealthy pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Readiness probe failed: Get "http://10.132.0.134:15021/healthz/ready": dial tcp 10.132.0.134:15021: connect: connection refused 36m Warning ProbeError pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Readiness probe error: Get "http://10.132.0.134:15021/healthz/ready": dial tcp 10.132.0.134:15021: connect: connection refused... 36m Normal Killing pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Stopping container istio-proxy 36m Warning ProbeError pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning ProbeError pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Startup probe error: Get "http://10.132.0.134:15021/healthz/ready": dial tcp 10.132.0.134:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-shared-b-w1-061bda-openshift-default-98c5f5dc9-bwp8t Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning Unhealthy pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Startup probe failed: Get "http://10.132.0.134:15021/healthz/ready": dial tcp 10.132.0.134:15021: connect: connection refused 36m Normal AddedLabel gateway/e2e-inf-b-11a0b0 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-11a0b0 36m Normal SuccessfulCreate replicaset/e2e-inf-b-11a0b0-openshift-default-6947dc8679 Created pod: e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 36m Normal ScalingReplicaSet deployment/e2e-inf-b-11a0b0-openshift-default Scaled up replica set e2e-inf-b-11a0b0-openshift-default-6947dc8679 from 0 to 1 36m Normal AddedInterface pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Add eth0 [10.132.0.142/23] from ovn-kubernetes 36m Normal Killing pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Stopping container payload-pre-processing 36m Normal Killing pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Stopping container payload-processing 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b Created pod: payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b Error creating: pods "payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-11a0b0: serviceaccount "payload-processing-e2e-inf-b-11a0b0" not found 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5 Created pod: payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5 Error creating: pods "payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-11a0b0: serviceaccount "payload-processing-e2e-inf-b-11a0b0" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-11a0b0 Scaled up replica set payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5 from 0 to 1 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-11a0b0 Scaled up replica set payload-processing-e2e-inf-b-11a0b0-fcbdbf85b from 0 to 1 36m Normal Pulled pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Add eth0 [10.132.0.143/23] from ovn-kubernetes 36m Normal Started pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Container started 36m Normal Created pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Container created 36m Normal Pulled pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Created pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Container created 36m Normal AddedInterface pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Add eth0 [10.132.0.144/23] from ovn-kubernetes 36m Normal Started pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Container started 35m Normal Pulled pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Container created 35m Normal Started pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Container started 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-5dc13e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-5dc13e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-65948c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-65948c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning Unhealthy pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Startup probe failed: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused 35m Warning ProbeError pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Startup probe error: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused... 35m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled down replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 3 to 2 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 2; reason: All metrics below target 35m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 35m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-5f9b95-649d6665ff-x7xxn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-processing-e2e-worker-w1-5f9b95-56c969bbb8-njmmb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-9hmf2 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default Scaled down replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 3 to 2 35m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 2; reason: All metrics below target 35m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-svjkt 35m Normal Killing pod/payload-processing-e2e-api-a-3f0723-5bf4b7db56-chr4h Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-api-a-3f0723-78b56d6d8c-mkwsm Stopping container payload-pre-processing 35m Normal Killing pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Stopping container payload-pre-processing 35m Normal Killing pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Stopping container payload-processing 35m Normal SuccessfulCreate replicaset/e2e-api-b-23491e-openshift-default-669fd4c48c Created pod: e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp 35m Normal ScalingReplicaSet deployment/e2e-api-b-23491e-openshift-default Scaled up replica set e2e-api-b-23491e-openshift-default-669fd4c48c from 0 to 1 35m Normal AddedLabel gateway/e2e-api-b-23491e Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-23491e 35m Normal AddedInterface pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Add eth0 [10.134.0.186/23] from ovn-kubernetes 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-5dc13e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-5dc13e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Normal Created pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Container created 35m Normal Started pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Container started 35m Normal Pulled pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-a78bf4-84bb7ff58c-242qr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-processing-e2e-worker-w5-a78bf4-857fb69748-rwsnw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-23491e Scaled up replica set payload-processing-e2e-api-b-23491e-7c7dc6dc6d from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-23491e-8566fddb9f Created pod: payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-23491e Scaled up replica set payload-pre-processing-e2e-api-b-23491e-8566fddb9f from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-23491e-7c7dc6dc6d Created pod: payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-23491e-8566fddb9f Error creating: pods "payload-pre-processing-e2e-api-b-23491e-8566fddb9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-23491e: serviceaccount "payload-processing-e2e-api-b-23491e" not found 35m Warning FailedCreate replicaset/payload-processing-e2e-api-b-23491e-7c7dc6dc6d Error creating: pods "payload-processing-e2e-api-b-23491e-7c7dc6dc6d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-23491e: serviceaccount "payload-processing-e2e-api-b-23491e" not found 35m Normal Created pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Container created 35m Normal Started pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Container started 35m Normal Pulled pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Pulled pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Add eth0 [10.132.0.147/23] from ovn-kubernetes 35m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Add eth0 [10.132.0.146/23] from ovn-kubernetes 35m Normal Created pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Container created 35m Normal Started pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Container started 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-cd2d96-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-cd2d96-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning Unhealthy pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Startup probe failed: Get "http://10.132.0.138:15021/healthz/ready": dial tcp 10.132.0.138:15021: connect: connection refused 35m Warning ProbeError pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Startup probe error: Get "http://10.132.0.138:15021/healthz/ready": dial tcp 10.132.0.138:15021: connect: connection refused... 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-23491e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-23491e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-cd2d96-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-cd2d96-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal AddedLabel gateway/e2e-api-b-92f229 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-92f229 34m Normal SuccessfulCreate replicaset/e2e-api-b-92f229-openshift-default-5d5584b4bc Created pod: e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj 34m Normal AddedInterface pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Add eth0 [10.132.0.149/23] from ovn-kubernetes 34m Normal ScalingReplicaSet deployment/e2e-api-b-92f229-openshift-default Scaled up replica set e2e-api-b-92f229-openshift-default-5d5584b4bc from 0 to 1 34m Normal Pulled pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Container started 34m Normal Created pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Container created 34m Warning FailedCreate replicaset/payload-processing-e2e-api-b-92f229-6bc5cb889b Error creating: pods "payload-processing-e2e-api-b-92f229-6bc5cb889b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-92f229: serviceaccount "payload-processing-e2e-api-b-92f229" not found 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-92f229 Scaled up replica set payload-processing-e2e-api-b-92f229-6bc5cb889b from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-92f229 Scaled up replica set payload-pre-processing-e2e-api-b-92f229-8579d8bc8c from 0 to 1 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c Error creating: pods "payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-92f229: serviceaccount "payload-processing-e2e-api-b-92f229" not found 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-11a0b0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c Created pod: payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-92f229-6bc5cb889b Created pod: payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-11a0b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal AddedInterface pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Add eth0 [10.132.0.150/23] from ovn-kubernetes 34m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Add eth0 [10.132.0.151/23] from ovn-kubernetes 34m Normal Pulled pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Pulled pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Container started 34m Normal Started pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Container started 34m Normal Created pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Container created 34m Normal Created pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Container created 34m Warning Unhealthy pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Startup probe failed: Get "http://10.132.0.142:15021/healthz/ready": dial tcp 10.132.0.142:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Startup probe error: Get "http://10.132.0.142:15021/healthz/ready": dial tcp 10.132.0.142:15021: connect: connection refused... 34m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Stopping container istio-proxy 34m Normal Killing pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Stopping container istio-proxy 34m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Readiness probe error: HTTP probe failed with statuscode: 503... 34m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-8xssf Readiness probe failed: HTTP probe failed with statuscode: 503 34m Warning Unhealthy pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Readiness probe failed: HTTP probe failed with statuscode: 503 34m Warning ProbeError pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 Readiness probe error: HTTP probe failed with statuscode: 503... 34m Normal Killing pod/e2e-mt-27219c73-openshift-default-88d599849-rnbxz Stopping container istio-proxy 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-11a0b0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-11a0b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal Created pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Container created 34m Normal Started pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Container started 34m Normal Pulled pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Killing pod/e2e-api-a-3f0723-openshift-default-898ff695-vl5s7 Stopping container istio-proxy 33m Warning Unhealthy pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Startup probe failed: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused 33m Warning ProbeError pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Startup probe error: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused... 33m Normal Killing pod/payload-processing-e2e-api-b-65948c-79bcdcd958-zmwjw Stopping container payload-processing 33m Normal Killing pod/payload-pre-processing-e2e-api-b-65948c-7f8c699d95-wlc4l Stopping container payload-pre-processing 33m Normal Killing pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Stopping container istio-proxy 33m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-b6sv7 33m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-fg56m 33m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-pr86v 33m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Add eth0 [10.132.0.154/23] from ovn-kubernetes 33m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Container started 33m Warning ProbeError pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Container created 33m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Add eth0 [10.132.0.153/23] from ovn-kubernetes 33m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Container created 33m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Container started 33m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Add eth0 [10.134.0.190/23] from ovn-kubernetes 33m Warning Unhealthy pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-5f7z2 Readiness probe failed: HTTP probe failed with statuscode: 503 33m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Container started 33m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Container created 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Startup probe failed: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Startup probe error: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused... 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Startup probe error: Get "http://10.132.0.154:15021/healthz/ready": dial tcp 10.132.0.154:15021: connect: connection refused... 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Startup probe failed: Get "http://10.132.0.154:15021/healthz/ready": dial tcp 10.132.0.154:15021: connect: connection refused 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Startup probe failed: Get "http://10.132.0.153:15021/healthz/ready": dial tcp 10.132.0.153:15021: connect: connection refused 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Startup probe error: Get "http://10.132.0.153:15021/healthz/ready": dial tcp 10.132.0.153:15021: connect: connection refused... 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-23491e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-23491e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal Started pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Container started 33m Normal Pulled pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Container created 33m Warning ProbeError pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Readiness probe error: Get "http://10.132.0.149:15021/healthz/ready": read tcp 10.132.0.2:37322->10.132.0.149:15021: read: connection reset by peer... 33m Warning Unhealthy pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Readiness probe failed: Get "http://10.132.0.149:15021/healthz/ready": read tcp 10.132.0.2:37322->10.132.0.149:15021: read: connection reset by peer 33m Warning Unhealthy pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Startup probe failed: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused 33m Warning ProbeError pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Startup probe error: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused... 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0a78ac-openshift-default New size: 1; reason: All metrics below target 33m Normal ScalingReplicaSet deployment/e2e-worker-w0-0a78ac-openshift-default Scaled down replica set e2e-worker-w0-0a78ac-openshift-default-854db56669 from 2 to 1 33m Normal SuccessfulDelete replicaset/e2e-worker-w0-0a78ac-openshift-default-854db56669 Deleted pod: e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd 33m Warning Unhealthy pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Readiness probe failed: HTTP probe failed with statuscode: 503 33m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Stopping container istio-proxy 33m Warning ProbeError pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal Killing pod/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf Stopping container payload-pre-processing 32m Normal Killing pod/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz Stopping container payload-processing 32m Normal Killing pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Stopping container payload-pre-processing 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-92f229-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-pzcsw Stopping container discovery 32m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-pzcsw 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-92f229-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 32m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-92f229-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-92f229-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-8e2fc1-65666cc8b7-ln74j Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-processing-e2e-worker-w4-8e2fc1-87b4cd784-fv8n8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Normal Killing pod/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Stopping container payload-pre-processing 32m Warning Unhealthy pod/payload-pre-processing-e2e-api-a-cd2d96-8c477746c-wpqmg Readiness probe failed: timeout: failed to connect service "10.132.0.139:50052" within 1s: context deadline exceeded 31m Normal Killing pod/e2e-api-b-65948c-openshift-default-b44f97585-gz54k Stopping container istio-proxy 31m Normal AddedLabel gateway/e2e-inf-a-2fa881 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-2fa881 31m Normal AddedInterface pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Add eth0 [10.134.0.205/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/e2e-inf-a-2fa881-openshift-default-67466498f9 Created pod: e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj 31m Normal ScalingReplicaSet deployment/e2e-inf-a-2fa881-openshift-default Scaled up replica set e2e-inf-a-2fa881-openshift-default-67466498f9 from 0 to 1 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-2fa881 Scaled up replica set payload-processing-e2e-inf-a-2fa881-579b754b68 from 0 to 1 31m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-2fa881-579b754b68 Error creating: pods "payload-processing-e2e-inf-a-2fa881-579b754b68-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2fa881: serviceaccount "payload-processing-e2e-inf-a-2fa881" not found 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-2fa881 Scaled up replica set payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f from 0 to 1 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f Error creating: pods "payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2fa881: serviceaccount "payload-processing-e2e-inf-a-2fa881" not found 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-2fa881-579b754b68 Created pod: payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 31m Normal Created pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Container created 31m Normal AddedInterface pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Add eth0 [10.134.0.206/23] from ovn-kubernetes 31m Normal Started pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Container started 31m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Add eth0 [10.134.0.207/23] from ovn-kubernetes 31m Normal Killing pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb Stopping container istio-proxy 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f Created pod: payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl 31m Normal Pulled pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Container started 31m Normal Pulled pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Created pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Container created 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-4ee72d-openshift-default New size: 1; reason: All metrics below target 31m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc Stopping container istio-proxy 31m Normal Killing pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-cpgmg Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/e2e-worker-w6-4ee72d-openshift-default-689c7db79 Deleted pod: e2e-worker-w6-4ee72d-openshift-default-689c7db79-59mfc 31m Normal ScalingReplicaSet deployment/e2e-worker-w6-4ee72d-openshift-default Scaled down replica set e2e-worker-w6-4ee72d-openshift-default-689c7db79 from 2 to 1 31m Normal Killing pod/payload-processing-e2e-api-b-23491e-7c7dc6dc6d-9zrfx Stopping container payload-processing 31m Normal Killing pod/payload-pre-processing-e2e-api-b-23491e-8566fddb9f-x55gf Stopping container payload-pre-processing 31m Normal Killing pod/e2e-api-a-cd2d96-openshift-default-5bb87dfdb5-p7p45 Stopping container istio-proxy 30m Normal Killing pod/payload-pre-processing-e2e-api-b-92f229-8579d8bc8c-9bd67 Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-api-b-92f229-6bc5cb889b-ck8p8 Stopping container payload-processing 30m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc Stopping container istio-proxy 30m Normal SuccessfulDelete replicaset/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c Deleted pod: e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-vv8zc 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-ae2d38-openshift-default New size: 1; reason: All metrics below target 30m Normal ScalingReplicaSet deployment/e2e-worker-w3-ae2d38-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c from 2 to 1 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2fa881-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Normal Created pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Container created 30m Normal Pulled pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2fa881-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Normal Started pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Container started 30m Warning ProbeError pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Startup probe error: Get "http://10.134.0.205:15021/healthz/ready": dial tcp 10.134.0.205:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Startup probe failed: Get "http://10.134.0.205:15021/healthz/ready": dial tcp 10.134.0.205:15021: connect: connection refused 30m Normal AddedLabel gateway/e2e-inf-b-a064b4 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-a064b4 30m Normal SuccessfulCreate replicaset/e2e-inf-b-a064b4-openshift-default-7d545fb5cf Created pod: e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv 30m Normal ScalingReplicaSet deployment/e2e-inf-b-a064b4-openshift-default Scaled up replica set e2e-inf-b-a064b4-openshift-default-7d545fb5cf from 0 to 1 30m Normal AddedInterface pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Add eth0 [10.134.0.210/23] from ovn-kubernetes 30m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-a064b4 Scaled up replica set payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95 from 0 to 1 30m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-a064b4 Scaled up replica set payload-processing-e2e-inf-b-a064b4-6c6fd49d56 from 0 to 1 30m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95 Error creating: pods "payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-a064b4: serviceaccount "payload-processing-e2e-inf-b-a064b4" not found 30m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95 Created pod: payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw 30m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-a064b4-6c6fd49d56 Error creating: pods "payload-processing-e2e-inf-b-a064b4-6c6fd49d56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-a064b4: serviceaccount "payload-processing-e2e-inf-b-a064b4" not found 30m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-a064b4-6c6fd49d56 Created pod: payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm 30m Normal Started pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Container started 30m Normal Pulled pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal Pulled pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Add eth0 [10.132.0.161/23] from ovn-kubernetes 30m Normal AddedInterface pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Add eth0 [10.132.0.160/23] from ovn-kubernetes 30m Normal Started pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Container started 30m Normal Created pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Container created 30m Normal Created pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Container created 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2fa881-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2fa881-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal Killing pod/e2e-api-b-23491e-openshift-default-669fd4c48c-nqsnp Stopping container istio-proxy 29m Normal Killing pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Stopping container payload-processing 29m Normal Killing pod/e2e-api-b-92f229-openshift-default-5d5584b4bc-wbjvj Stopping container istio-proxy 29m Warning Unhealthy pod/payload-processing-e2e-worker-w2-df031b-7f46bd6f58-h8snx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-df031b-6f59bb8759-htbxl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal Killing pod/payload-processing-e2e-worker-w3-ae2d38-d5c76b6db-qhfpw Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-worker-w3-ae2d38-5c8446448c-mjcq6 Stopping container payload-pre-processing 28m Normal Pulled pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Started pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Container started 28m Normal Created pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Container created 28m Normal Killing pod/payload-processing-e2e-inf-a-bcef0e-6b58d4f4d6-kch8q Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Stopping container payload-pre-processing 28m Warning ProbeError pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Startup probe error: Get "http://10.134.0.210:15021/healthz/ready": dial tcp 10.134.0.210:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Startup probe failed: Get "http://10.134.0.210:15021/healthz/ready": dial tcp 10.134.0.210:15021: connect: connection refused 28m Normal Killing pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-82nv9 Stopping container istio-proxy 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-a064b4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-a064b4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-bcef0e-677f8dc679-t5gql Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-a064b4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-a064b4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Normal Killing pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Stopping container istio-proxy 28m Warning ProbeError pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning Unhealthy pod/e2e-inf-a-bcef0e-openshift-default-58c977d9fc-n8kjv Readiness probe failed: HTTP probe failed with statuscode: 503 27m Normal Killing pod/payload-processing-e2e-inf-b-11a0b0-fcbdbf85b-pccqh Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Stopping container payload-pre-processing 27m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-11a0b0-6777dddff5-7qz5q Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/e2e-inf-b-11a0b0-openshift-default-6947dc8679-rq8s6 Stopping container istio-proxy 27m Normal Killing pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Stopping container payload-processing 27m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-pdvns 27m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-pr86v Stopping container istio-proxy 27m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 10 to 5 27m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Stopping container istio-proxy 27m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-pr86v 27m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Stopping container istio-proxy 27m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-fg56m Stopping container istio-proxy 27m Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-7cs6l Stopping container istio-proxy 27m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-fg56m 27m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-7cs6l 27m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 27m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-kwkj5 27m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-kwkj5 Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-pdvns Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-0a78ac-74b759bb59-pvlkt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-processing-e2e-worker-w0-0a78ac-f8dd4dc46-p482k Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Normal Killing pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-vflpj Stopping container istio-proxy 24m Normal Killing pod/payload-processing-e2e-inf-a-2fa881-579b754b68-fw8r6 Stopping container payload-processing 24m Normal Killing pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Stopping container payload-pre-processing 24m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-2fa881-6fdb85cc9f-2qlbl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/e2e-inf-a-2fa881-openshift-default-67466498f9-gbmqj Stopping container istio-proxy 24m Normal Killing pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Stopping container payload-processing 24m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-a064b4-64fcc5bf95-5ltgw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning Unhealthy pod/payload-processing-e2e-inf-b-a064b4-6c6fd49d56-zwpdm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/e2e-inf-b-a064b4-openshift-default-7d545fb5cf-s74jv Stopping container istio-proxy 23m Normal Killing pod/payload-pre-processing-e2e-worker-w6-4ee72d-5b8b8d9787-mpwgg Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-worker-w6-4ee72d-766db79cc6-bvp6r Stopping container payload-processing 23m Normal Killing pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Stopping container istio-proxy 23m Warning Unhealthy pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw Readiness probe error: HTTP probe failed with statuscode: 503... 22m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-d2bvn 22m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-f8t7j 22m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 5 to 7 22m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Add eth0 [10.134.0.233/23] from ovn-kubernetes 22m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Container started 22m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Container created 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Add eth0 [10.134.0.232/23] from ovn-kubernetes 22m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Container started 22m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Container created 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Startup probe failed: Get "http://10.134.0.232:15021/healthz/ready": dial tcp 10.134.0.232:15021: connect: connection refused 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Startup probe error: Get "http://10.134.0.232:15021/healthz/ready": dial tcp 10.134.0.232:15021: connect: connection refused... 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Startup probe error: Get "http://10.134.0.233:15021/healthz/ready": dial tcp 10.134.0.233:15021: connect: connection refused... 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-d2bvn Startup probe failed: Get "http://10.134.0.233:15021/healthz/ready": dial tcp 10.134.0.233:15021: connect: connection refused 22m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 22m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 7 to 10 22m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-tjm7q 22m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Container started 22m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Container started 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Add eth0 [10.132.0.166/23] from ovn-kubernetes 22m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Container created 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Add eth0 [10.134.0.235/23] from ovn-kubernetes 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Add eth0 [10.132.0.165/23] from ovn-kubernetes 22m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Container created 22m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Container started 22m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Container created 22m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Startup probe error: Get "http://10.134.0.235:15021/healthz/ready": dial tcp 10.134.0.235:15021: connect: connection refused... 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Startup probe failed: Get "http://10.134.0.235:15021/healthz/ready": dial tcp 10.134.0.235:15021: connect: connection refused 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Startup probe failed: Get "http://10.132.0.166:15021/healthz/ready": dial tcp 10.132.0.166:15021: connect: connection refused 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Startup probe failed: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Startup probe error: Get "http://10.132.0.166:15021/healthz/ready": dial tcp 10.132.0.166:15021: connect: connection refused... 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Startup probe error: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused... 21m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Readiness probe error: Get "http://10.132.0.153:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Readiness probe failed: Get "http://10.134.0.235:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-xbx2q Readiness probe error: Get "http://10.134.0.235:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Readiness probe error: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-tjm7q Readiness probe failed: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Readiness probe error: Get "http://10.132.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Readiness probe error: Get "http://10.134.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-b6sv7 Readiness probe failed: Get "http://10.132.0.153:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-sf6r9 Readiness probe failed: Get "http://10.132.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-f8t7j Readiness probe failed: Get "http://10.134.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 15m Normal AddedLabel gateway/e2e-worker-single-main-819d1b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-819d1b 15m Normal ScalingReplicaSet deployment/e2e-worker-single-main-819d1b-openshift-default Scaled up replica set e2e-worker-single-main-819d1b-openshift-default-b6b946f45 from 0 to 1 15m Normal SuccessfulCreate replicaset/e2e-worker-single-main-819d1b-openshift-default-b6b946f45 Created pod: e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt 15m Normal AddedInterface pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Add eth0 [10.134.0.251/23] from ovn-kubernetes 15m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-819d1b Scaled up replica set payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5b7bd7 from 0 to 1 15m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-819d1b Scaled up replica set payload-processing-e2e-worker-single-main-819d1b-766fb7f458 from 0 to 1 15m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5b7bd7 Error creating: pods "payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5b7bd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-819d1b: serviceaccount "payload-processing-e2e-worker-single-main-819d1b" not found 15m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5b7bd7 Created pod: payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg 15m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-819d1b-766fb7f458 Error creating: pods "payload-processing-e2e-worker-single-main-819d1b-766fb7f458-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-819d1b: serviceaccount "payload-processing-e2e-worker-single-main-819d1b" not found 15m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-819d1b-766fb7f458 Created pod: payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r 15m Normal Created pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Container created 15m Normal Started pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Container started 15m Normal Created pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Container created 15m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Add eth0 [10.132.0.169/23] from ovn-kubernetes 15m Normal Pulled pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Add eth0 [10.132.0.168/23] from ovn-kubernetes 15m Normal Started pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Container started 15m Normal Pulled pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 15m Normal Started pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Container started 15m Normal Created pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Container created 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-819d1b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 15m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-819d1b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning Unhealthy pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Startup probe failed: Get "http://10.134.0.251:15021/healthz/ready": dial tcp 10.134.0.251:15021: connect: connection refused 14m Warning ProbeError pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Startup probe error: Get "http://10.134.0.251:15021/healthz/ready": dial tcp 10.134.0.251:15021: connect: connection refused... 14m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-819d1b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 14m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-819d1b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 13m Normal Killing pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Stopping container payload-processing 13m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Stopping container payload-pre-processing 13m Normal Killing pod/e2e-worker-single-main-819d1b-openshift-default-b6b946f45-469bt Stopping container istio-proxy 13m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-819d1b-5f9c5dm9rg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 13m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-819d1b-766fb7f45mrf4r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 11m Normal AddedLabel gateway/e2e-worker-single-main-87bb59 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-87bb59 11m Normal AddedInterface pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Add eth0 [10.134.1.2/23] from ovn-kubernetes 11m Normal SuccessfulCreate replicaset/e2e-worker-single-main-87bb59-openshift-default-6b98645df7 Created pod: e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j 11m Normal ScalingReplicaSet deployment/e2e-worker-single-main-87bb59-openshift-default Scaled up replica set e2e-worker-single-main-87bb59-openshift-default-6b98645df7 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-87bb59 Scaled up replica set payload-pre-processing-e2e-worker-single-main-87bb59-c5dcccdf9 from 0 to 1 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4b Error creating: pods "payload-processing-e2e-worker-single-main-87bb59-544cc9cc4b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-87bb59: serviceaccount "payload-processing-e2e-worker-single-main-87bb59" not found 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-87bb59-c5dcccdf9 Error creating: pods "payload-pre-processing-e2e-worker-single-main-87bb59-c5dcccdf9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-87bb59: serviceaccount "payload-processing-e2e-worker-single-main-87bb59" not found 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-87bb59 Scaled up replica set payload-processing-e2e-worker-single-main-87bb59-544cc9cc4b from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4b Created pod: payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj 10m Normal Started pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Container started 10m Normal Created pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Container created 10m Normal Pulled pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Add eth0 [10.132.0.172/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-87bb59-c5dcccdf9 Created pod: payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh 10m Normal Started pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Container started 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Add eth0 [10.132.0.173/23] from ovn-kubernetes 10m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Created pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Container created 10m Normal Pulled pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Normal Started pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Container started 10m Normal Created pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Container created 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-87bb59-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-87bb59-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning ProbeError pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Startup probe error: Get "http://10.134.1.2:15021/healthz/ready": dial tcp 10.134.1.2:15021: connect: connection refused... 10m Warning Unhealthy pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Startup probe failed: Get "http://10.134.1.2:15021/healthz/ready": dial tcp 10.134.1.2:15021: connect: connection refused 9m47s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-87bb59-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m47s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-87bb59-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m20s Normal Killing pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Stopping container payload-processing 9m20s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Stopping container payload-pre-processing 9m6s Normal Killing pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Stopping container istio-proxy 9m6s Warning FailedToRetrieveImagePullSecret pod/e2e-worker-single-main-87bb59-openshift-default-6b98645df7q482j Unable to retrieve some image pull secrets (e2e-worker-single-main-87bb59-openshift-default-dockercfg-cs884); attempting to pull the image may not succeed. 9m2s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-87bb59-544cc9cc4wdjhj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 9m2s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-87bb59-c5dccpc9lh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m38s Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Readiness probe failed: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m38s Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll Readiness probe error: Get "http://10.133.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m22s Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-695mh Readiness probe error: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m22s Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-695mh Readiness probe failed: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m56s Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Readiness probe failed: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m56s Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Readiness probe error: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m56s Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-mr976 Readiness probe failed: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m56s Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-k7458 Readiness probe error: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m26s Normal AddedLabel gateway/e2e-ait-fb4770d7 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-fb4770d7 6m26s Normal SuccessfulCreate replicaset/e2e-ait-fb4770d7-openshift-default-9bc6d99b7 Created pod: e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 6m26s Normal ScalingReplicaSet deployment/e2e-ait-fb4770d7-openshift-default Scaled up replica set e2e-ait-fb4770d7-openshift-default-9bc6d99b7 from 0 to 1 6m25s Normal AddedInterface pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Add eth0 [10.134.1.10/23] from ovn-kubernetes 6m19s Warning FailedCreate replicaset/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d Error creating: pods "payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-fb4770d7: serviceaccount "payload-processing-e2e-ait-fb4770d7" not found 6m19s Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-fb4770d7 Scaled up replica set payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d from 0 to 1 6m19s Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9 Error creating: pods "payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-fb4770d7: serviceaccount "payload-processing-e2e-ait-fb4770d7" not found 6m19s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-fb4770d7 Scaled up replica set payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9 from 0 to 1 6m18s Normal AddedInterface pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Add eth0 [10.132.0.180/23] from ovn-kubernetes 6m18s Normal Started pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Container started 6m18s Normal Pulled pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m18s Normal AddedInterface pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Add eth0 [10.132.0.178/23] from ovn-kubernetes 6m18s Normal Pulled pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m18s Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d Created pod: payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg 6m18s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9 Created pod: payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv 6m18s Normal Created pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Container created 6m17s Normal Started pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Container started 6m17s Normal Created pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Container created 6m1s Normal Pulled pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m Normal Created pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Container created 6m Normal Started pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Container started 5m55s Warning Unhealthy pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Startup probe failed: Get "http://10.134.1.10:15021/healthz/ready": dial tcp 10.134.1.10:15021: connect: connection refused 5m55s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-fb4770d7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m55s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-fb4770d7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m55s Warning ProbeError pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Startup probe error: Get "http://10.134.1.10:15021/healthz/ready": dial tcp 10.134.1.10:15021: connect: connection refused... 5m34s Normal Killing pod/payload-processing-e2e-ait-fb4770d7-6f8cdc8f4d-2p5qg Stopping container payload-processing 5m34s Normal Killing pod/payload-pre-processing-e2e-ait-fb4770d7-66c958c8f9-qs5vv Stopping container payload-pre-processing 5m25s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-fb4770d7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m25s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-fb4770d7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m23s Normal Killing pod/e2e-ait-fb4770d7-openshift-default-9bc6d99b7-dbtv2 Stopping container istio-proxy 5m22s Normal ScalingReplicaSet deployment/e2e-authcfg-http-0a3a62-openshift-default Scaled up replica set e2e-authcfg-http-0a3a62-openshift-default-5666fb7897 from 0 to 1 5m22s Normal SuccessfulCreate replicaset/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897 Created pod: e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn 5m22s Normal AddedLabel gateway/e2e-authcfg-http-0a3a62 Added label istio.io/rev=openshift-gateway to gateway e2e-authcfg-http-0a3a62 5m21s Normal AddedInterface pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Add eth0 [10.134.1.12/23] from ovn-kubernetes 5m14s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc Created pod: payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl 5m14s Warning FailedCreate replicaset/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8 Error creating: pods "payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62: serviceaccount "payload-processing-e2e-authcfg-http-0a3a62" not found 5m14s Normal SuccessfulCreate replicaset/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8 Created pod: payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v 5m14s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-authcfg-http-0a3a62 Scaled up replica set payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc from 0 to 1 5m14s Warning FailedCreate replicaset/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc Error creating: pods "payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-0a3a62: serviceaccount "payload-processing-e2e-authcfg-http-0a3a62" not found 5m14s Normal ScalingReplicaSet deployment/payload-processing-e2e-authcfg-http-0a3a62 Scaled up replica set payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8 from 0 to 1 5m13s Normal Started pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Container started 5m13s Normal Pulled pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m13s Normal Created pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Container created 5m13s Normal AddedInterface pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Add eth0 [10.132.0.181/23] from ovn-kubernetes 5m13s Normal Pulled pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m13s Normal Created pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Container created 5m13s Normal AddedInterface pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Add eth0 [10.132.0.182/23] from ovn-kubernetes 5m13s Normal Started pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Container started 4m55s Normal Created pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Container created 4m55s Normal Pulled pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m55s Normal Started pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Container started 4m50s Warning ProbeError pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Startup probe error: Get "http://10.134.1.12:15021/healthz/ready": dial tcp 10.134.1.12:15021: connect: connection refused... 4m50s Warning Unhealthy pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Startup probe failed: Get "http://10.134.1.12:15021/healthz/ready": dial tcp 10.134.1.12:15021: connect: connection refused 4m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-0a3a62-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-0a3a62-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 3m20s Normal Killing pod/payload-pre-processing-e2e-authcfg-http-0a3a62-7c494646bc-85lkl Stopping container payload-pre-processing 3m20s Normal Killing pod/payload-processing-e2e-authcfg-http-0a3a62-5f6bf546f8-2z89v Stopping container payload-processing 3m1s Normal Killing pod/e2e-authcfg-http-0a3a62-openshift-default-5666fb7897-jh7cn Stopping container istio-proxy