=== events in openshift-ingress (2026-10-30T06:41:26Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 62m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Successfully assigned openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh to ip-10-0-136-35.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Successfully assigned openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 to ip-10-0-138-15.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Successfully assigned openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj to ip-10-0-136-35.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Successfully assigned openshift-ingress/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq to ip-10-0-136-173.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Successfully assigned openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Successfully assigned openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st to ip-10-0-136-35.ec2.internal 22m Normal Scheduled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Successfully assigned openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Successfully assigned openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 to ip-10-0-138-15.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Successfully assigned openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Successfully assigned openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx to ip-10-0-136-173.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Successfully assigned openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs to ip-10-0-136-35.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Successfully assigned openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Successfully assigned openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp to ip-10-0-138-15.ec2.internal 48m Normal Scheduled pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Successfully assigned openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 to ip-10-0-136-35.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Successfully assigned openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg to ip-10-0-138-15.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Successfully assigned openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb to ip-10-0-138-15.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Successfully assigned openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv to ip-10-0-136-35.ec2.internal 9m3s Normal Scheduled pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Successfully assigned openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 to ip-10-0-138-15.ec2.internal 5m30s Normal Scheduled pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Successfully assigned openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq to ip-10-0-138-15.ec2.internal 28m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct to ip-10-0-136-173.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x to ip-10-0-136-173.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 to ip-10-0-138-15.ec2.internal 21m Normal Scheduled pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Successfully assigned openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 to ip-10-0-136-35.ec2.internal 54m Normal Scheduled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Successfully assigned openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Successfully assigned openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 to ip-10-0-138-15.ec2.internal 25m Normal Scheduled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Successfully assigned openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 to ip-10-0-136-173.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Successfully assigned openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Successfully assigned openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v to ip-10-0-138-15.ec2.internal 54m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 to ip-10-0-136-35.ec2.internal 31m Normal Scheduled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Successfully assigned openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz to ip-10-0-136-35.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 to ip-10-0-136-35.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj to ip-10-0-138-15.ec2.internal 36m Normal Scheduled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Successfully assigned openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 to ip-10-0-136-35.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 to ip-10-0-136-35.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Successfully assigned openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw to ip-10-0-136-173.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr to ip-10-0-138-15.ec2.internal 70m Normal Scheduled pod/router-default-59857694c5-wq9fj Successfully assigned openshift-ingress/router-default-59857694c5-wq9fj to ip-10-0-136-173.ec2.internal 50m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f to ip-10-0-138-15.ec2.internal 42m Normal Scheduled pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Successfully assigned openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 to ip-10-0-136-35.ec2.internal 70m Warning FailedScheduling pod/router-default-59857694c5-wq9fj no nodes available to schedule pods 76m Warning FailedScheduling pod/router-default-59857694c5-wq9fj no nodes available to schedule pods 53m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v to ip-10-0-136-173.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 to ip-10-0-138-15.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Successfully assigned openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx to ip-10-0-136-35.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Successfully assigned openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 to ip-10-0-138-15.ec2.internal 56m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm to ip-10-0-138-15.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Successfully assigned openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr to ip-10-0-136-173.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 to ip-10-0-138-15.ec2.internal 36m Normal Scheduled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Successfully assigned openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks to ip-10-0-136-173.ec2.internal 46m Normal Scheduled pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Successfully assigned openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb to ip-10-0-138-15.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Successfully assigned openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx to ip-10-0-136-173.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z to ip-10-0-136-35.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Successfully assigned openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Successfully assigned openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h to ip-10-0-136-35.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Successfully assigned openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b to ip-10-0-136-173.ec2.internal 44m Normal Scheduled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Successfully assigned openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp to ip-10-0-136-35.ec2.internal 44m Warning FailedScheduling pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 45m Warning FailedScheduling pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 45m Normal Preempted pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Preempted by pod 6792a38e-911d-4406-8c1a-30bf2bbfddb8 on node ip-10-0-136-173.ec2.internal 46m Normal Scheduled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Successfully assigned openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Successfully assigned openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 to ip-10-0-136-35.ec2.internal 49m Normal Scheduled pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Successfully assigned openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx to ip-10-0-136-35.ec2.internal 25m Normal Scheduled pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Successfully assigned openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t to ip-10-0-138-15.ec2.internal 50m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x to ip-10-0-138-15.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Successfully assigned openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh to ip-10-0-136-173.ec2.internal 51m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh to ip-10-0-138-15.ec2.internal 44m Warning FailedScheduling pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Warning FailedScheduling pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Successfully assigned openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z to ip-10-0-136-35.ec2.internal 23m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf to ip-10-0-138-15.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw to ip-10-0-136-173.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf to ip-10-0-136-35.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Successfully assigned openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms to ip-10-0-136-35.ec2.internal 50m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf to ip-10-0-138-15.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 to ip-10-0-138-15.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg to ip-10-0-136-173.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Successfully assigned openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n to ip-10-0-136-173.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 to ip-10-0-138-15.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Successfully assigned openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 to ip-10-0-136-35.ec2.internal 44m Warning FailedScheduling pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Warning FailedScheduling pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Successfully assigned openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp to ip-10-0-138-15.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr to ip-10-0-138-15.ec2.internal 17m Normal Scheduled pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Successfully assigned openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k to ip-10-0-136-173.ec2.internal 54m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd to ip-10-0-136-173.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 to ip-10-0-136-35.ec2.internal 5m25s Normal Scheduled pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c to ip-10-0-136-35.ec2.internal 8m58s Normal Scheduled pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj to ip-10-0-136-35.ec2.internal 30m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 to ip-10-0-136-173.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn to ip-10-0-138-15.ec2.internal 50m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Successfully assigned openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw to ip-10-0-138-15.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 to ip-10-0-138-15.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw to ip-10-0-136-35.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc to ip-10-0-136-173.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z to ip-10-0-136-35.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm to ip-10-0-136-173.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz to ip-10-0-138-15.ec2.internal 23m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn to ip-10-0-138-15.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Successfully assigned openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 to ip-10-0-136-173.ec2.internal 50m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Successfully assigned openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf to ip-10-0-138-15.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Successfully assigned openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb to ip-10-0-136-35.ec2.internal 21m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv to ip-10-0-138-15.ec2.internal 20m Normal Scheduled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Successfully assigned openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n to ip-10-0-136-173.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Successfully assigned openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Successfully assigned openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn to ip-10-0-136-35.ec2.internal 50m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 to ip-10-0-138-15.ec2.internal 27m Normal Scheduled pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Successfully assigned openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 to ip-10-0-136-35.ec2.internal 49m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 to ip-10-0-138-15.ec2.internal 50m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm to ip-10-0-136-173.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Successfully assigned openshift-ingress/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj to ip-10-0-136-173.ec2.internal 29m Normal Scheduled pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Successfully assigned openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 to ip-10-0-138-15.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp to ip-10-0-138-15.ec2.internal 26m Normal Scheduled pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Successfully assigned openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Successfully assigned openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs to ip-10-0-136-173.ec2.internal 19m Normal Scheduled pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Successfully assigned openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 to ip-10-0-136-35.ec2.internal 44m Normal Scheduled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Successfully assigned openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z to ip-10-0-136-35.ec2.internal 50m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 to ip-10-0-136-173.ec2.internal 63m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-fcmfp Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp to ip-10-0-136-35.ec2.internal 38m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v to ip-10-0-138-15.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r to ip-10-0-136-35.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Successfully assigned openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz to ip-10-0-136-173.ec2.internal 30m Normal Scheduled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Successfully assigned openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 to ip-10-0-136-173.ec2.internal 41m Normal Scheduled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Successfully assigned openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Successfully assigned openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 to ip-10-0-138-15.ec2.internal 45m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv 0/3 nodes are available: 3 Insufficient cpu. 23m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c to ip-10-0-136-173.ec2.internal 45m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: not eligible due to a terminating pod on the nominated node. 38m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 to ip-10-0-136-173.ec2.internal 45m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: not eligible due to a terminating pod on the nominated node. 56m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Successfully assigned openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm to ip-10-0-138-15.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Successfully assigned openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z to ip-10-0-136-35.ec2.internal 45m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6rrhv to ip-10-0-136-173.ec2.internal 48m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d54kn to ip-10-0-136-173.ec2.internal 31m Normal Scheduled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Successfully assigned openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 to ip-10-0-136-35.ec2.internal 63m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 to ip-10-0-136-173.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Successfully assigned openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj to ip-10-0-136-35.ec2.internal 51m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk to ip-10-0-138-15.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn to ip-10-0-136-173.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Successfully assigned openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t to ip-10-0-136-35.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Successfully assigned openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m to ip-10-0-136-35.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp to ip-10-0-136-35.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 to ip-10-0-136-35.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 to ip-10-0-138-15.ec2.internal 21m Normal Scheduled pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 to ip-10-0-136-173.ec2.internal 38m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q to ip-10-0-136-35.ec2.internal 16m Normal Scheduled pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 to ip-10-0-138-15.ec2.internal 17m Normal Scheduled pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt to ip-10-0-138-15.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn to ip-10-0-136-35.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg to ip-10-0-136-173.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 to ip-10-0-136-35.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l to ip-10-0-136-35.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr to ip-10-0-138-15.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj to ip-10-0-136-173.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 to ip-10-0-136-35.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn to ip-10-0-136-35.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq to ip-10-0-136-35.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx to ip-10-0-136-35.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 to ip-10-0-136-173.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb to ip-10-0-138-15.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd to ip-10-0-136-35.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 to ip-10-0-136-35.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm to ip-10-0-136-35.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg to ip-10-0-136-35.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Successfully assigned openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc to ip-10-0-138-15.ec2.internal 45m Warning FailedScheduling pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 45m Warning FailedScheduling pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 40m Normal Scheduled pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Successfully assigned openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-blgbh to ip-10-0-136-173.ec2.internal 63m Normal Scheduled pod/kube-auth-proxy-8596b47b87-7k748 Successfully assigned openshift-ingress/kube-auth-proxy-8596b47b87-7k748 to ip-10-0-136-35.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc to ip-10-0-136-35.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m to ip-10-0-136-35.ec2.internal 31m Normal Scheduled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Successfully assigned openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf to ip-10-0-138-15.ec2.internal 50m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn to ip-10-0-136-35.ec2.internal 63m Normal Scheduled pod/kube-auth-proxy-8596b47b87-pk7dd Successfully assigned openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Successfully assigned openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Successfully assigned openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd to ip-10-0-138-15.ec2.internal 33m Normal Scheduled pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Successfully assigned openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 to ip-10-0-138-15.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt to ip-10-0-136-173.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j to ip-10-0-138-15.ec2.internal 50m Normal Scheduled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Successfully assigned openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl to ip-10-0-136-173.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 to ip-10-0-138-15.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 to ip-10-0-136-35.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 to ip-10-0-136-35.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Successfully assigned openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz to ip-10-0-138-15.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5cp4f to ip-10-0-136-173.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp to ip-10-0-138-15.ec2.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z to ip-10-0-138-15.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd to ip-10-0-136-35.ec2.internal 19m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 to ip-10-0-136-35.ec2.internal 33m Normal Scheduled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Successfully assigned openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl to ip-10-0-136-173.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Successfully assigned openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc to ip-10-0-136-35.ec2.internal 34m Normal Scheduled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Successfully assigned openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7 to ip-10-0-136-35.ec2.internal 57m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Successfully assigned openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 to ip-10-0-138-15.ec2.internal 60m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk to ip-10-0-138-15.ec2.internal 32m Normal Scheduled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Successfully assigned openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv to ip-10-0-136-173.ec2.internal 22m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d to ip-10-0-138-15.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Successfully assigned openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh to ip-10-0-136-35.ec2.internal 33m Normal Scheduled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Successfully assigned openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 to ip-10-0-136-173.ec2.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-n57mc to ip-10-0-136-173.ec2.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p99cc to ip-10-0-136-173.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Successfully assigned openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b to ip-10-0-136-35.ec2.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qgv7r to ip-10-0-136-173.ec2.internal 57m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Successfully assigned openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 to ip-10-0-138-15.ec2.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Successfully assigned openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd to ip-10-0-138-15.ec2.internal 57m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfps7 to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xjhtt to ip-10-0-136-173.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Successfully assigned openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg to ip-10-0-136-173.ec2.internal 40m Normal Scheduled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Successfully assigned openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n to ip-10-0-138-15.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Successfully assigned openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn to ip-10-0-136-35.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-7b98bb678c-dlwvz Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz to ip-10-0-138-15.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn to ip-10-0-138-15.ec2.internal 57m Normal Scheduled pod/payload-processing-55fbbbf9dc-bq9hh Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-bq9hh to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq to ip-10-0-136-35.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Successfully assigned openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 to ip-10-0-136-35.ec2.internal 38m Warning FailedScheduling pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 38m Warning FailedScheduling pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 47m Normal Scheduled pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Successfully assigned openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Successfully assigned openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr to ip-10-0-136-35.ec2.internal 28m Normal Scheduled pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Successfully assigned openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 to ip-10-0-138-15.ec2.internal 51m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn to ip-10-0-138-15.ec2.internal 30m Normal Scheduled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Successfully assigned openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Successfully assigned openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw to ip-10-0-138-15.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc to ip-10-0-136-173.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Successfully assigned openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs to ip-10-0-138-15.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj to ip-10-0-136-35.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b to ip-10-0-136-35.ec2.internal 35m Normal Scheduled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Successfully assigned openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt to ip-10-0-136-35.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg to ip-10-0-138-15.ec2.internal 34m Normal Scheduled pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Successfully assigned openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 to ip-10-0-136-35.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Successfully assigned openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn to ip-10-0-138-15.ec2.internal 31m Normal Scheduled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Successfully assigned openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 to ip-10-0-136-35.ec2.internal 30m Normal Scheduled pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Successfully assigned openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq to ip-10-0-138-15.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm to ip-10-0-138-15.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql to ip-10-0-136-35.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f to ip-10-0-136-35.ec2.internal 36m Normal Scheduled pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Successfully assigned openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk to ip-10-0-136-35.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc to ip-10-0-138-15.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 to ip-10-0-136-35.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk to ip-10-0-138-15.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m to ip-10-0-136-35.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j to ip-10-0-138-15.ec2.internal 41m Normal Scheduled pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Successfully assigned openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp to ip-10-0-136-35.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm to ip-10-0-136-35.ec2.internal 41m Normal Scheduled pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Successfully assigned openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs to ip-10-0-136-35.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d to ip-10-0-138-15.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 to ip-10-0-138-15.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 to ip-10-0-136-35.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk to ip-10-0-138-15.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj to ip-10-0-136-173.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k to ip-10-0-138-15.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt to ip-10-0-136-35.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 to ip-10-0-136-35.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl to ip-10-0-136-35.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl to ip-10-0-138-15.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Successfully assigned openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 to ip-10-0-138-15.ec2.internal 20m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr to ip-10-0-136-35.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv to ip-10-0-136-35.ec2.internal 5m25s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 to ip-10-0-138-15.ec2.internal 8m58s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl to ip-10-0-138-15.ec2.internal 30m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 to ip-10-0-138-15.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Successfully assigned openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt to ip-10-0-136-173.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p to ip-10-0-136-35.ec2.internal 17m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 to ip-10-0-138-15.ec2.internal 16m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 to ip-10-0-138-15.ec2.internal 21m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 to ip-10-0-136-35.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 to ip-10-0-138-15.ec2.internal 19m Normal Scheduled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Successfully assigned openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 to ip-10-0-136-173.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 to ip-10-0-136-35.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Successfully assigned openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl to ip-10-0-136-35.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp to ip-10-0-136-35.ec2.internal 76m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-59857694c5 from 0 to 1 76m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 76m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 76m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 76m Normal SuccessfulCreate replicaset/router-default-59857694c5 Created pod: router-default-59857694c5-wq9fj 70m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 69m Warning FailedMount pod/router-default-59857694c5-wq9fj MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 69m Warning FailedMount pod/router-default-59857694c5-wq9fj MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 69m Warning FailedMount pod/router-default-59857694c5-wq9fj MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 69m Normal AddedInterface pod/router-default-59857694c5-wq9fj Add eth0 [10.132.0.11/23] from ovn-kubernetes 69m Normal Pulled pod/router-default-59857694c5-wq9fj Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 69m Normal Created pod/router-default-59857694c5-wq9fj Container created 69m Normal Started pod/router-default-59857694c5-wq9fj Container started 63m Normal SuccessfulCreate replicaset/kube-auth-proxy-8596b47b87 Created pod: kube-auth-proxy-8596b47b87-7k748 63m Normal SuccessfulCreate replicaset/kube-auth-proxy-8596b47b87 Created pod: kube-auth-proxy-8596b47b87-pk7dd 63m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-8596b47b87 from 0 to 2 63m Normal Pulling pod/kube-auth-proxy-8596b47b87-7k748 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 63m Normal AddedInterface pod/kube-auth-proxy-8596b47b87-7k748 Add eth0 [10.133.0.22/23] from ovn-kubernetes 63m Normal AddedInterface pod/kube-auth-proxy-8596b47b87-pk7dd Add eth0 [10.134.0.17/23] from ovn-kubernetes 63m Normal Pulling pod/kube-auth-proxy-8596b47b87-pk7dd Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 63m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 63m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-fcmfp 63m Normal Created pod/kube-auth-proxy-8596b47b87-7k748 Container created 63m Normal Pulled pod/kube-auth-proxy-8596b47b87-pk7dd Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.473s (2.473s including waiting). Image size: 188904677 bytes. 63m Normal Created pod/kube-auth-proxy-8596b47b87-pk7dd Container created 63m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-fcmfp Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 63m Normal Started pod/kube-auth-proxy-8596b47b87-7k748 Container started 63m Normal Pulled pod/kube-auth-proxy-8596b47b87-7k748 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.251s (2.251s including waiting). Image size: 188904677 bytes. 63m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-fcmfp Add eth0 [10.133.0.23/23] from ovn-kubernetes 63m Normal Started pod/kube-auth-proxy-8596b47b87-pk7dd Container started 63m Normal Started pod/istiod-openshift-gateway-7876bfcb74-fcmfp Container started 63m Normal Created pod/istiod-openshift-gateway-7876bfcb74-fcmfp Container created 63m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-fcmfp Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.824s (2.824s including waiting). Image size: 185476477 bytes. 63m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Add eth0 [10.132.0.40/23] from ovn-kubernetes 63m Normal Pulling pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 63m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-59cf4dc6bf from 0 to 1 63m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-59cf4dc6bf Created pod: data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 63m Normal Started pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Container started 63m Normal Created pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Container created 63m Normal Pulled pod/data-science-gateway-data-science-gateway-class-59cf4dc6bfjxkq2 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.409s (2.409s including waiting). Image size: 183555333 bytes. 63m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Container created 62m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 62m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 62m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Container started 62m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Add eth0 [10.133.0.26/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-rdsvq 62m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Readiness probe error: HTTP probe failed with statuscode: 503... 62m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq Readiness probe failed: HTTP probe failed with statuscode: 503 62m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 62m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-fcmfp Stopping container discovery 62m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-fcmfp 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-96dmr 60m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 60m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Add eth0 [10.133.0.33/23] from ovn-kubernetes 60m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 0 to 1 60m Normal Pulling pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 60m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.648s (2.648s including waiting). Image size: 183555333 bytes. 58m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Container created 58m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Container started 58m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Startup probe error: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-xfps7 57m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 1 to 2 57m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Add eth0 [10.132.0.54/23] from ovn-kubernetes 57m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Container created 57m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Container started 57m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Startup probe failed: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused 57m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Startup probe error: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused... 57m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1 57m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 57m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 57m Normal SuccessfulCreate replicaset/payload-processing-55fbbbf9dc Created pod: payload-processing-55fbbbf9dc-bq9hh 57m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-dlwvz 57m Warning FailedCreate replicaset/payload-processing-55fbbbf9dc Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 57m Normal Pulling pod/payload-pre-processing-7b98bb678c-dlwvz Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 57m Normal AddedInterface pod/payload-processing-55fbbbf9dc-bq9hh Add eth0 [10.133.0.41/23] from ovn-kubernetes 57m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-dlwvz Add eth0 [10.134.0.32/23] from ovn-kubernetes 57m Normal Pulling pod/payload-processing-55fbbbf9dc-bq9hh Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 57m Normal Pulled pod/payload-processing-55fbbbf9dc-bq9hh Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.854s (1.854s including waiting). Image size: 125317101 bytes. 57m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Add eth0 [10.134.0.33/23] from ovn-kubernetes 57m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 57m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 2 to 4 57m Normal Created pod/payload-processing-55fbbbf9dc-bq9hh Container created 57m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-8r2sh 57m Normal Started pod/payload-pre-processing-7b98bb678c-dlwvz Container started 57m Normal Created pod/payload-pre-processing-7b98bb678c-dlwvz Container created 57m Normal Pulled pod/payload-pre-processing-7b98bb678c-dlwvz Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.938s (1.938s including waiting). Image size: 125317101 bytes. 57m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-rxgds 57m Normal Started pod/payload-processing-55fbbbf9dc-bq9hh Container started 57m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Add eth0 [10.133.0.42/23] from ovn-kubernetes 57m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Container started 57m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Container created 57m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Pulling pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 57m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.764s (2.764s including waiting). Image size: 183555333 bytes. 57m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Container created 57m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Container started 57m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused... 57m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused 57m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Startup probe failed: Get "http://10.134.0.33:15021/healthz/ready": dial tcp 10.134.0.33:15021: connect: connection refused 57m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Startup probe error: Get "http://10.134.0.33:15021/healthz/ready": dial tcp 10.134.0.33:15021: connect: connection refused... 56m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-p99cc 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-xjhtt 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-84wh7 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-7dz5z 56m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 4 to 8 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Add eth0 [10.134.0.34/23] from ovn-kubernetes 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Add eth0 [10.133.0.44/23] from ovn-kubernetes 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Add eth0 [10.132.0.56/23] from ovn-kubernetes 56m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Container created 56m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Container created 56m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Add eth0 [10.132.0.55/23] from ovn-kubernetes 56m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Container created 56m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Container started 56m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Container started 56m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Container started 56m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Container started 56m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Container created 56m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Startup probe error: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Startup probe failed: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused... 56m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Startup probe error: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh 56m Normal AddedLabel gateway/e2e-worker-w1-4a68a1 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-4a68a1 56m Normal AddedLabel gateway/e2e-worker-w5-a35835 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-a35835 56m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc 56m Normal AddedLabel gateway/e2e-worker-w3-128436 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-128436 56m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 0 to 1 56m Normal AddedLabel gateway/e2e-worker-w0-197e9b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-197e9b 56m Normal AddedLabel gateway/e2e-worker-w2-c78066 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-c78066 56m Normal AddedLabel gateway/e2e-worker-w6-1354a5 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-1354a5 56m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled up replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 0 to 1 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled up replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 0 to 1 56m Normal AddedLabel gateway/e2e-worker-w4-e2e3d0 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-e2e3d0 56m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf 56m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Add eth0 [10.133.0.48/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Add eth0 [10.133.0.47/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm 56m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 0 to 1 56m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 0 to 1 56m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Add eth0 [10.134.0.37/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Add eth0 [10.133.0.46/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Add eth0 [10.134.0.35/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp 56m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 0 to 1 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled up replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 0 to 1 56m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn 56m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Add eth0 [10.134.0.36/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Add eth0 [10.134.0.38/23] from ovn-kubernetes 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5 Error creating: pods "payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-c78066: serviceaccount "payload-processing-e2e-worker-w2-c78066" not found 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Add eth0 [10.133.0.49/23] from ovn-kubernetes 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-c78066 Scaled up replica set payload-pre-processing-e2e-worker-w2-c78066-6b57567c95 from 0 to 1 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95 Created pod: payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5 Created pod: payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-c78066 Scaled up replica set payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5 from 0 to 1 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95 Error creating: pods "payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-c78066: serviceaccount "payload-processing-e2e-worker-w2-c78066" not found 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Created pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Container created 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-197e9b Scaled up replica set payload-processing-e2e-worker-w0-197e9b-6bd8477b49 from 0 to 1 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-197e9b Scaled up replica set payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879 from 0 to 1 56m Normal Pulled pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-197e9b-6bd8477b49 Error creating: pods "payload-processing-e2e-worker-w0-197e9b-6bd8477b49-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-197e9b: serviceaccount "payload-processing-e2e-worker-w0-197e9b" not found 56m Normal AddedInterface pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Add eth0 [10.133.0.50/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879 Created pod: payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879 Error creating: pods "payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-197e9b: serviceaccount "payload-processing-e2e-worker-w0-197e9b" not found 56m Normal Created pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Container created 56m Normal Started pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Container started 56m Normal Started pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Container started 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-197e9b-6bd8477b49 Created pod: payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-a35835 Scaled up replica set payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc from 0 to 1 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Pulled pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Created pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Container created 56m Normal Started pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Container started 56m Normal Started pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Container started 56m Normal Created pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Container created 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc Error creating: pods "payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-a35835: serviceaccount "payload-processing-e2e-worker-w5-a35835" not found 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Add eth0 [10.133.0.51/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc Created pod: payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-a35835 Scaled up replica set payload-processing-e2e-worker-w5-a35835-9765b9fcf from 0 to 1 56m Normal AddedInterface pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Add eth0 [10.133.0.52/23] from ovn-kubernetes 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-a35835-9765b9fcf Error creating: pods "payload-processing-e2e-worker-w5-a35835-9765b9fcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-a35835: serviceaccount "payload-processing-e2e-worker-w5-a35835" not found 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-a35835-9765b9fcf Created pod: payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h 56m Normal Created pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Container created 56m Normal Started pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Container started 56m Normal Created pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Container created 56m Normal Started pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Container started 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Add eth0 [10.133.0.54/23] from ovn-kubernetes 56m Normal AddedInterface pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Add eth0 [10.133.0.53/23] from ovn-kubernetes 56m Normal Pulled pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5 Created pod: payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c Created pod: payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5 Error creating: pods "payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0: serviceaccount "payload-processing-e2e-worker-w4-e2e3d0" not found 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c Error creating: pods "payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0: serviceaccount "payload-processing-e2e-worker-w4-e2e3d0" not found 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-e2e3d0 Scaled up replica set payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c from 0 to 1 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-e2e3d0 Scaled up replica set payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5 from 0 to 1 56m Normal Pulled pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Started pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Container started 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-4a68a1-7fc787db87 Error creating: pods "payload-processing-e2e-worker-w1-4a68a1-7fc787db87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-4a68a1: serviceaccount "payload-processing-e2e-worker-w1-4a68a1" not found 56m Normal Started pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Container started 56m Normal AddedInterface pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Add eth0 [10.133.0.55/23] from ovn-kubernetes 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-4a68a1-7fc787db87 Created pod: payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f Created pod: payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p 56m Normal Created pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Container created 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-4a68a1 Scaled up replica set payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f from 0 to 1 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-4a68a1 Scaled up replica set payload-processing-e2e-worker-w1-4a68a1-7fc787db87 from 0 to 1 56m Normal Created pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Container created 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Add eth0 [10.133.0.56/23] from ovn-kubernetes 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f Error creating: pods "payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-4a68a1: serviceaccount "payload-processing-e2e-worker-w1-4a68a1" not found 56m Normal Pulled pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Started pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Container started 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c Error creating: pods "payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-1354a5: serviceaccount "payload-processing-e2e-worker-w6-1354a5" not found 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-1354a5 Scaled up replica set payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99 from 0 to 1 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99 Created pod: payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t 56m Normal Created pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Container created 56m Normal Created pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Container created 56m Normal Started pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Container started 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Add eth0 [10.133.0.57/23] from ovn-kubernetes 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99 Error creating: pods "payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-1354a5: serviceaccount "payload-processing-e2e-worker-w6-1354a5" not found 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-1354a5 Scaled up replica set payload-processing-e2e-worker-w6-1354a5-5b6cc9875c from 0 to 1 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c Created pod: payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Add eth0 [10.134.0.41/23] from ovn-kubernetes 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-128436-d84dd445f Error creating: pods "payload-pre-processing-e2e-worker-w3-128436-d84dd445f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-128436: serviceaccount "payload-processing-e2e-worker-w3-128436" not found 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-128436-79ddc6fd69 Error creating: pods "payload-processing-e2e-worker-w3-128436-79ddc6fd69-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-128436: serviceaccount "payload-processing-e2e-worker-w3-128436" not found 56m Normal Pulled pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Add eth0 [10.134.0.42/23] from ovn-kubernetes 56m Normal Created pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Container created 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Add eth0 [10.133.0.58/23] from ovn-kubernetes 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-128436 Scaled up replica set payload-processing-e2e-worker-w3-128436-79ddc6fd69 from 0 to 1 56m Normal Started pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Container started 56m Normal Started pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Container started 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-128436 Scaled up replica set payload-pre-processing-e2e-worker-w3-128436-d84dd445f from 0 to 1 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-128436-79ddc6fd69 Created pod: payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-128436-d84dd445f Created pod: payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk 56m Normal Created pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Container created 56m Normal Pulled pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Created pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Container created 56m Normal Started pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Container started 56m Normal Started pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Container started 56m Normal AddedInterface pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Add eth0 [10.133.0.60/23] from ovn-kubernetes 56m Normal Created pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Container created 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Add eth0 [10.133.0.59/23] from ovn-kubernetes 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Readiness probe error: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Readiness probe failed: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Container started 56m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Container created 55m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Container started 55m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Container created 55m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Container created 55m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Container started 55m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Startup probe failed: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Startup probe error: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused... 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Container started 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Container created 55m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Container started 55m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Container created 55m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Startup probe failed: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Startup probe error: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Readiness probe failed: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Readiness probe error: Get "http://10.134.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Container started 55m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Container started 55m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused 55m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Container created 55m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused... 55m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Container created 55m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 1 to 2 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Container created 55m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Add eth0 [10.134.0.47/23] from ovn-kubernetes 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Container started 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Startup probe failed: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Startup probe error: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused... 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv 54m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled up replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 1 to 2 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Container started 54m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Add eth0 [10.133.0.65/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Container created 54m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Startup probe failed: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Startup probe error: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused... 54m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl 54m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled up replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 1 to 2 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Add eth0 [10.133.0.66/23] from ovn-kubernetes 54m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 1 to 2 54m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Container created 54m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Container started 54m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Add eth0 [10.134.0.48/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Container started 54m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Container created 54m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Startup probe failed: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Startup probe error: Get "http://10.134.0.48:15021/healthz/ready": dial tcp 10.134.0.48:15021: connect: connection refused... 54m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 1 to 2 54m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 1 to 2 54m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 54m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Container started 54m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Add eth0 [10.134.0.49/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Container started 54m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Container created 54m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Add eth0 [10.133.0.68/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Container created 54m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Startup probe failed: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Startup probe error: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused... 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 2 to 4 54m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Container started 54m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Container created 54m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Add eth0 [10.132.0.68/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Add eth0 [10.133.0.73/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Container created 54m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Container started 53m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw 53m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 2 to 3 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled up replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 1 to 2 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw 53m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Container created 53m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Container started 53m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Add eth0 [10.132.0.69/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Container created 53m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Add eth0 [10.133.0.81/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Container started 53m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Readiness probe failed: Get "http://10.132.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Readiness probe error: Get "http://10.132.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Readiness probe failed: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Readiness probe error: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 53m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Add eth0 [10.134.0.51/23] from ovn-kubernetes 53m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 (combined from similar events): Created pod: maas-default-gateway-openshift-default-58847f9fc5-2c9x4 53m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Container created 53m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Container started 53m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Add eth0 [10.133.0.85/23] from ovn-kubernetes 53m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Container created 53m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Container started 53m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-dvbqk 53m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 8 to 10 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn 53m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Add eth0 [10.134.0.52/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Add eth0 [10.132.0.70/23] from ovn-kubernetes 53m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 4 to 6 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct 53m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Container started 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz 53m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled up replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 2 to 3 53m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Container created 53m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Container started 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Add eth0 [10.132.0.71/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Container created 53m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Container created 53m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Container started 53m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Startup probe error: Get "http://10.134.0.51:15021/healthz/ready": dial tcp 10.134.0.51:15021: connect: connection refused... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Startup probe failed: Get "http://10.134.0.51:15021/healthz/ready": dial tcp 10.134.0.51:15021: connect: connection refused 53m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Startup probe error: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Startup probe failed: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused 53m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 2 to 3 53m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 3 to 4 53m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 53m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Container created 53m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Add eth0 [10.134.0.54/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Add eth0 [10.132.0.72/23] from ovn-kubernetes 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd 53m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Add eth0 [10.133.0.87/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Container started 53m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Container started 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 2 to 4 53m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Container started 53m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Container created 53m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Container started 53m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Container created 53m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Container created 53m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Add eth0 [10.132.0.73/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Startup probe failed: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Startup probe failed: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Startup probe error: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Startup probe error: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 4 to 5 52m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Container started 52m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Container created 52m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Add eth0 [10.133.0.89/23] from ovn-kubernetes 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z 52m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Startup probe error: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Startup probe failed: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused 52m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Add eth0 [10.132.0.75/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Add eth0 [10.133.0.94/23] from ovn-kubernetes 52m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 6 to 8 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 5 to 6 52m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Add eth0 [10.132.0.76/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Container created 52m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Container created 52m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Container started 52m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Container started 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 52m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Container started 52m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Container created 52m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Startup probe error: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Startup probe failed: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn 51m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled up replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 8 to 10 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 51m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Add eth0 [10.134.0.59/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c (combined from similar events): Created pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Container started 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Container started 51m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Add eth0 [10.134.0.61/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Container created 51m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled up replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 6 to 10 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Container started 51m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Container created 51m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Container started 51m Normal Started pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz 51m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Add eth0 [10.132.0.78/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 (combined from similar events): Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Container created 51m Normal AddedInterface pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Add eth0 [10.133.0.100/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Container created 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Add eth0 [10.132.0.77/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Add eth0 [10.134.0.60/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Created pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Add eth0 [10.133.0.101/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 3 to 6 51m Normal Created pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Container created 51m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Container started 51m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Container started 51m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Container created 51m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Container started 51m Normal Started pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Container started 51m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Container created 51m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Container created 51m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Container created 51m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Add eth0 [10.133.0.103/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Add eth0 [10.133.0.102/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Startup probe failed: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Startup probe error: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Startup probe failed: Get "http://10.133.0.100:15021/healthz/ready": dial tcp 10.133.0.100:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Startup probe error: Get "http://10.133.0.100:15021/healthz/ready": dial tcp 10.133.0.100:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Startup probe error: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Startup probe error: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Startup probe failed: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Startup probe failed: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Startup probe error: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Startup probe failed: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled up replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 2 to 4 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t 51m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled up replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 3 to 5 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z 51m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs 51m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn 51m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Add eth0 [10.134.0.63/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Container created 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Add eth0 [10.134.0.64/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Container started 51m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Container created 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 51m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Container created 51m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled up replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 2 to 4 51m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Add eth0 [10.134.0.62/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Add eth0 [10.132.0.81/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Container created 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Container created 51m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Container started 51m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Add eth0 [10.132.0.82/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Add eth0 [10.132.0.80/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Add eth0 [10.133.0.107/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Container started 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 9; reason: All metrics below target 51m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 6 to 9 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Container started 51m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Add eth0 [10.133.0.109/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Container created 51m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Container started 51m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Container created 51m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Container started 51m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Container started 51m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Container created 51m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Container created 51m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Container started 51m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Add eth0 [10.133.0.108/23] from ovn-kubernetes 51m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Startup probe error: Get "http://10.133.0.109:15021/healthz/ready": dial tcp 10.133.0.109:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Startup probe failed: Get "http://10.133.0.109:15021/healthz/ready": dial tcp 10.133.0.109:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Startup probe error: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Startup probe failed: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Startup probe failed: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Startup probe failed: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Startup probe error: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Startup probe error: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Startup probe failed: Get "http://10.133.0.108:15021/healthz/ready": dial tcp 10.133.0.108:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Startup probe error: Get "http://10.133.0.108:15021/healthz/ready": dial tcp 10.133.0.108:15021: connect: connection refused... 51m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 51m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-xfmtk 51m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 51m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Add eth0 [10.134.0.65/23] from ovn-kubernetes 51m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Readiness probe error: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 51m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Readiness probe failed: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Container started 51m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.014s (3.014s including waiting). Image size: 185476477 bytes. 51m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Container created 51m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Readiness probe failed: HTTP probe failed with statuscode: 503 51m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Readiness probe error: HTTP probe failed with statuscode: 503... 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf 51m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Add eth0 [10.132.0.86/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Container created 51m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Container started 51m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Add eth0 [10.134.0.66/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Add eth0 [10.133.0.116/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Container created 51m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Container started 51m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Container created 51m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Container started 51m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Container created 51m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Add eth0 [10.132.0.87/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 4 to 8 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh 51m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Startup probe failed: Get "http://10.133.0.116:15021/healthz/ready": dial tcp 10.133.0.116:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Startup probe error: Get "http://10.133.0.116:15021/healthz/ready": dial tcp 10.133.0.116:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Startup probe failed: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Startup probe error: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f 50m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled up replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 5 to 10 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff (combined from similar events): Created pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 50m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Container created 50m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Container created 50m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Container started 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled up replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 9 to 10 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Add eth0 [10.132.0.89/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Container created 50m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Container created 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 (combined from similar events): Created pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv 50m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Container started 50m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Add eth0 [10.134.0.67/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Container started 50m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Add eth0 [10.133.0.122/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Container created 50m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Add eth0 [10.133.0.121/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Container started 50m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm 50m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled up replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 4 to 6 50m Normal AddedInterface pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Add eth0 [10.132.0.88/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Container started 50m Normal Created pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Container created 50m Normal Started pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Container started 50m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Add eth0 [10.132.0.90/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Add eth0 [10.133.0.123/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Container started 50m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Add eth0 [10.134.0.68/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Container started 50m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Container created 50m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Container created 50m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Readiness probe error: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Readiness probe failed: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Readiness probe failed: Get "http://10.133.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Readiness probe failed: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Readiness probe failed: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Readiness probe error: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Readiness probe failed: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Readiness probe error: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Readiness probe failed: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Readiness probe failed: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Readiness probe error: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Readiness probe error: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Readiness probe failed: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Readiness probe failed: Get "http://10.134.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Readiness probe error: Get "http://10.133.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Readiness probe error: Get "http://10.134.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Readiness probe failed: Get "http://10.133.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Readiness probe error: Get "http://10.134.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Readiness probe error: Get "http://10.133.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Readiness probe error: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Readiness probe failed: Get "http://10.132.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Readiness probe failed: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Readiness probe error: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Readiness probe failed: Get "http://10.132.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Readiness probe error: Get "http://10.134.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Readiness probe error: Get "http://10.132.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Readiness probe failed: Get "http://10.134.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Readiness probe error: Get "http://10.132.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Readiness probe failed: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Readiness probe error: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Readiness probe error: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Readiness probe failed: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Readiness probe failed: Get "http://10.133.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Readiness probe error: Get "http://10.133.0.100:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Readiness probe failed: Get "http://10.133.0.100:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Readiness probe error: Get "http://10.133.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Readiness probe failed: Get "http://10.133.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Readiness probe error: Get "http://10.133.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Startup probe failed: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Startup probe failed: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Startup probe error: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Startup probe failed: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Startup probe error: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Startup probe failed: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Startup probe error: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Startup probe failed: Get "http://10.133.0.123:15021/healthz/ready": dial tcp 10.133.0.123:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Startup probe error: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Startup probe error: Get "http://10.133.0.123:15021/healthz/ready": dial tcp 10.133.0.123:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c (combined from similar events): Created pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled up replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 8 to 10 50m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Add eth0 [10.134.0.69/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Add eth0 [10.133.0.132/23] from ovn-kubernetes 50m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Readiness probe failed: Get "http://10.133.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Readiness probe error: Get "http://10.133.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal Pulled pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Container started 50m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Container created 50m Normal Created pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Container created 50m Normal Started pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Container started 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Startup probe error: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Startup probe failed: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Startup probe failed: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Startup probe error: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Readiness probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Readiness probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Readiness probe failed: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Readiness probe error: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Readiness probe error: Get "http://10.133.0.107:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Readiness probe failed: Get "http://10.133.0.107:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal ScalingReplicaSet deployment/e2e-migrate-54d4943b-gw-openshift-default Scaled up replica set e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f from 0 to 1 49m Normal AddedLabel gateway/e2e-migrate-54d4943b-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-54d4943b-gw 49m Normal SuccessfulCreate replicaset/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f Created pod: e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx 49m Normal AddedInterface pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Add eth0 [10.133.0.141/23] from ovn-kubernetes 49m Normal Created pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Container created 49m Normal Pulled pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Container started 49m Normal AddedLabel gateway/e2e-ait-709c4831 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-709c4831 49m Normal Created pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Container created 49m Normal Pulled pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal SuccessfulCreate replicaset/e2e-ait-709c4831-openshift-default-647b55f95f Created pod: e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq 49m Normal AddedInterface pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Add eth0 [10.132.0.93/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/e2e-ait-709c4831-openshift-default Scaled up replica set e2e-ait-709c4831-openshift-default-647b55f95f from 0 to 1 49m Normal Started pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Container started 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Container created 49m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Add eth0 [10.133.0.143/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Container started 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Add eth0 [10.134.0.70/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Container created 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Container started 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Add eth0 [10.132.0.94/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Container created 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Container started 49m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 49m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Container started 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Container created 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Add eth0 [10.133.0.142/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 (combined from similar events): Created pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d 49m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled up replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 4 to 8 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled up replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 6 to 10 49m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Container created 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-54d4943b-8587c75899 Created pod: payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-54d4943b Scaled up replica set payload-pre-processing-e2e-migrate-54d4943b-77458c6b99 from 0 to 1 49m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Add eth0 [10.132.0.95/23] from ovn-kubernetes 49m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Container created 49m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-54d4943b Scaled up replica set payload-processing-e2e-migrate-54d4943b-8587c75899 from 0 to 1 49m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Container created 49m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Add eth0 [10.132.0.96/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Container started 49m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Container started 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99 Created pod: payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99 Error creating: pods "payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-54d4943b: serviceaccount "payload-processing-e2e-migrate-54d4943b" not found 49m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Container started 49m Normal Pulled pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Container started 49m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Add eth0 [10.134.0.71/23] from ovn-kubernetes 49m Normal AddedInterface pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Add eth0 [10.133.0.144/23] from ovn-kubernetes 49m Normal Created pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Container created 49m Warning FailedCreate replicaset/payload-processing-e2e-migrate-54d4943b-8587c75899 Error creating: pods "payload-processing-e2e-migrate-54d4943b-8587c75899-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-54d4943b: serviceaccount "payload-processing-e2e-migrate-54d4943b" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-709c4831-7f7c6c9465 Created pod: payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-709c4831-d78d7899f Created pod: payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn 49m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Add eth0 [10.133.0.145/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Container created 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-709c4831 Scaled up replica set payload-processing-e2e-ait-709c4831-7f7c6c9465 from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-709c4831 Scaled up replica set payload-pre-processing-e2e-ait-709c4831-d78d7899f from 0 to 1 49m Normal Created pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Container created 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-709c4831-d78d7899f Error creating: pods "payload-pre-processing-e2e-ait-709c4831-d78d7899f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-709c4831: serviceaccount "payload-processing-e2e-ait-709c4831" not found 49m Normal AddedInterface pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Add eth0 [10.133.0.146/23] from ovn-kubernetes 49m Normal Pulled pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Warning FailedCreate replicaset/payload-processing-e2e-ait-709c4831-7f7c6c9465 Error creating: pods "payload-processing-e2e-ait-709c4831-7f7c6c9465-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-709c4831: serviceaccount "payload-processing-e2e-ait-709c4831" not found 49m Normal Started pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Container started 49m Normal Started pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Container started 49m Normal Created pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Container created 49m Normal AddedInterface pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Add eth0 [10.134.0.72/23] from ovn-kubernetes 49m Normal Started pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Container started 49m Normal Pulled pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Container created 49m Normal AddedInterface pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Add eth0 [10.134.0.73/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Container started 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Startup probe error: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Startup probe failed: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Startup probe failed: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Startup probe error: Get "http://10.133.0.142:15021/healthz/ready": dial tcp 10.133.0.142:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Startup probe failed: Get "http://10.133.0.142:15021/healthz/ready": dial tcp 10.133.0.142:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Startup probe error: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Startup probe error: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Startup probe failed: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Startup probe error: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Startup probe failed: Get "http://10.133.0.143:15021/healthz/ready": dial tcp 10.133.0.143:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Startup probe error: Get "http://10.133.0.143:15021/healthz/ready": dial tcp 10.133.0.143:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Startup probe failed: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Startup probe failed: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Startup probe error: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused... 49m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled up replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 8 to 10 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Add eth0 [10.132.0.97/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Container started 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Container created 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 (combined from similar events): Created pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 10; reason: All metrics below target 49m Normal Started pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Container started 49m Normal AddedInterface pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Add eth0 [10.133.0.149/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Container created 49m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Readiness probe failed: Get "http://10.133.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Readiness probe error: Get "http://10.133.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Readiness probe error: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Readiness probe error: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Readiness probe failed: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Readiness probe failed: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Readiness probe error: Get "http://10.132.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Readiness probe failed: Get "http://10.132.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Readiness probe error: Get "http://10.133.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Readiness probe failed: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Readiness probe error: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Readiness probe error: Get "http://10.133.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Readiness probe failed: Get "http://10.133.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Readiness probe failed: Get "http://10.133.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Readiness probe error: Get "http://10.132.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Readiness probe failed: Get "http://10.132.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Readiness probe error: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Readiness probe failed: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Readiness probe failed: Get "http://10.133.0.103:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Readiness probe error: Get "http://10.133.0.103:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-709c4831-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Startup probe failed: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-709c4831-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 49m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Startup probe error: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused... 49m Normal AddedLabel gateway/e2e-shared-a-w3-6b5b1e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-6b5b1e 49m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-6b5b1e-openshift-default Scaled up replica set e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 from 0 to 1 49m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 Created pod: e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 49m Normal AddedInterface pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Add eth0 [10.134.0.75/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499 Created pod: payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e: serviceaccount "payload-processing-e2e-shared-a-w3-6b5b1e" not found 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-6b5b1e Scaled up replica set payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65 from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65 Error creating: pods "payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e: serviceaccount "payload-processing-e2e-shared-a-w3-6b5b1e" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65 Created pod: payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-6b5b1e Scaled up replica set payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499 from 0 to 1 49m Normal Pulled pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Container started 49m Normal Created pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Container created 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Add eth0 [10.133.0.152/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Container created 49m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Container started 49m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Add eth0 [10.133.0.151/23] from ovn-kubernetes 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-54d4943b-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-54d4943b-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal Killing pod/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47 Stopping container payload-pre-processing 48m Normal Killing pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Stopping container payload-processing 48m Normal Killing pod/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd Stopping container payload-processing 48m Normal Killing pod/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn Stopping container payload-pre-processing 48m Warning Unhealthy pod/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m Readiness probe failed: timeout: failed to connect service "10.133.0.146:50052" within 1s: context deadline exceeded 48m Normal Killing pod/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx Stopping container istio-proxy 48m Warning ProbeError pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Readiness probe error: Get "http://10.132.0.93:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Warning Unhealthy pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Readiness probe failed: Get "http://10.132.0.93:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Normal AddedLabel gateway/e2e-ait-a095fc35 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-a095fc35 48m Normal SuccessfulCreate replicaset/e2e-ait-a095fc35-openshift-default-574ccf8d8f Created pod: e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 48m Normal ScalingReplicaSet deployment/e2e-ait-a095fc35-openshift-default Scaled up replica set e2e-ait-a095fc35-openshift-default-574ccf8d8f from 0 to 1 48m Normal Killing pod/e2e-ait-709c4831-openshift-default-647b55f95f-w6zlq Stopping container istio-proxy 48m Normal AddedInterface pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Add eth0 [10.133.0.155/23] from ovn-kubernetes 48m Normal AddedLabel gateway/e2e-derive-e2a1565f Added label istio.io/rev=openshift-gateway to gateway e2e-derive-e2a1565f 48m Normal SuccessfulCreate replicaset/e2e-derive-e2a1565f-openshift-default-7cc9875d49 Created pod: e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj 48m Normal ScalingReplicaSet deployment/e2e-derive-e2a1565f-openshift-default Scaled up replica set e2e-derive-e2a1565f-openshift-default-7cc9875d49 from 0 to 1 48m Normal Started pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Container started 48m Normal Created pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Container created 48m Normal AddedInterface pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Add eth0 [10.132.0.99/23] from ovn-kubernetes 48m Normal Pulled pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 48m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 48m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-d54kn 48m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 48m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Add eth0 [10.132.0.100/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-a095fc35 Scaled up replica set payload-processing-e2e-ait-a095fc35-54d6b699 from 0 to 1 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9 Error creating: pods "payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-a095fc35: serviceaccount "payload-processing-e2e-ait-a095fc35" not found 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9 Created pod: payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-a095fc35 Scaled up replica set payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9 from 0 to 1 48m Warning FailedCreate replicaset/payload-processing-e2e-ait-a095fc35-54d6b699 Error creating: pods "payload-processing-e2e-ait-a095fc35-54d6b699-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-a095fc35: serviceaccount "payload-processing-e2e-ait-a095fc35" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-a095fc35-54d6b699 Created pod: payload-processing-e2e-ait-a095fc35-54d6b699-9brgh 48m Normal AddedInterface pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Add eth0 [10.133.0.157/23] from ovn-kubernetes 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9 Error creating: pods "payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-e2a1565f: serviceaccount "payload-processing-e2e-derive-e2a1565f" not found 48m Normal Pulled pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Container created 48m Normal Started pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Container started 48m Normal Pulled pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Container created 48m Normal Started pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Container started 48m Normal AddedInterface pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Add eth0 [10.133.0.156/23] from ovn-kubernetes 48m Warning FailedCreate replicaset/payload-processing-e2e-derive-e2a1565f-6c755fd957 Error creating: pods "payload-processing-e2e-derive-e2a1565f-6c755fd957-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-e2a1565f: serviceaccount "payload-processing-e2e-derive-e2a1565f" not found 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-e2a1565f Scaled up replica set payload-processing-e2e-derive-e2a1565f-6c755fd957 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-e2a1565f Scaled up replica set payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9 from 0 to 1 48m Normal Pulled pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Container created 48m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.797s (2.797s including waiting). Image size: 185476477 bytes. 48m Normal Started pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Container started 48m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Container started 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-e2a1565f-6c755fd957 Created pod: payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9 Created pod: payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg 48m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Container created 48m Normal AddedInterface pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Add eth0 [10.134.0.78/23] from ovn-kubernetes 48m Normal Started pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Container started 48m Normal Created pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Container created 48m Normal Pulled pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Readiness probe error: HTTP probe failed with statuscode: 503... 48m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Readiness probe failed: HTTP probe failed with statuscode: 503 48m Normal AddedInterface pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Add eth0 [10.134.0.77/23] from ovn-kubernetes 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-a095fc35-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-a095fc35-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled up replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 48m Normal AddedLabel gateway/e2e-shared-b-w3-c73fcb Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-c73fcb 48m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Add eth0 [10.133.0.158/23] from ovn-kubernetes 48m Normal Started pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Container started 48m Normal Created pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Container created 48m Normal Pulled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77 Error creating: pods "payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb: serviceaccount "payload-processing-e2e-shared-b-w3-c73fcb" not found 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-c73fcb Scaled up replica set payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-c73fcb Scaled up replica set payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655 from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655 Created pod: payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655 Error creating: pods "payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb: serviceaccount "payload-processing-e2e-shared-b-w3-c73fcb" not found 48m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Add eth0 [10.133.0.159/23] from ovn-kubernetes 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77 Created pod: payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt 48m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Container started 48m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Add eth0 [10.133.0.160/23] from ovn-kubernetes 48m Normal Created pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Container created 48m Normal Pulled pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Container created 48m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Container started 48m Warning ProbeError pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Startup probe error: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused... 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-e2a1565f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-e2a1565f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning Unhealthy pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Startup probe failed: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused 47m Warning Unhealthy pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Readiness probe failed: Get "http://10.133.0.155:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 47m Warning ProbeError pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Readiness probe error: Get "http://10.133.0.155:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 47m Normal SuccessfulCreate replicaset/e2e-mt-1b2243ef-openshift-default-88dfb6485 Created pod: e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm 47m Normal AddedLabel gateway/e2e-mt-1b2243ef Added label istio.io/rev=openshift-gateway to gateway e2e-mt-1b2243ef 47m Normal ScalingReplicaSet deployment/e2e-mt-1b2243ef-openshift-default Scaled up replica set e2e-mt-1b2243ef-openshift-default-88dfb6485 from 0 to 1 47m Normal AddedInterface pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Add eth0 [10.133.0.163/23] from ovn-kubernetes 47m Normal Pulled pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Container created 47m Normal Started pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Container started 47m Normal Killing pod/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh Stopping container payload-processing 47m Normal Killing pod/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k Stopping container payload-processing 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf Error creating: pods "payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1b2243ef: serviceaccount "payload-processing-e2e-mt-1b2243ef" not found 47m Warning ProbeError pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Startup probe error: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Startup probe failed: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf Created pod: payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-1b2243ef Scaled up replica set payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf from 0 to 1 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-a095fc35-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-a095fc35-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedCreate replicaset/payload-processing-e2e-mt-1b2243ef-7955fbbb49 Error creating: pods "payload-processing-e2e-mt-1b2243ef-7955fbbb49-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1b2243ef: serviceaccount "payload-processing-e2e-mt-1b2243ef" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-1b2243ef-7955fbbb49 Created pod: payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-1b2243ef Scaled up replica set payload-processing-e2e-mt-1b2243ef-7955fbbb49 from 0 to 1 47m Normal AddedInterface pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Add eth0 [10.134.0.83/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-477f6b-openshift-default Scaled up replica set e2e-shared-a-w6-477f6b-openshift-default-7854c78755 from 0 to 1 47m Normal Pulled pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-477f6b-openshift-default-7854c78755 Created pod: e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l 47m Normal Created pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Container created 47m Normal Started pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Container started 47m Normal Created pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Container created 47m Normal Pulled pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedLabel gateway/e2e-shared-a-w6-477f6b Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-477f6b 47m Normal Started pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Container started 47m Normal AddedInterface pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Add eth0 [10.134.0.82/23] from ovn-kubernetes 47m Normal AddedInterface pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Add eth0 [10.133.0.165/23] from ovn-kubernetes 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Killing pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 Stopping container istio-proxy 47m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-815776-openshift-default Scaled up replica set e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9 from 0 to 1 47m Normal AddedLabel gateway/e2e-shared-a-w4-815776 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w4-815776 47m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9 Created pod: e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r 47m Normal AddedInterface pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Add eth0 [10.134.0.84/23] from ovn-kubernetes 47m Normal Killing pod/e2e-derive-e2a1565f-openshift-default-7cc9875d49-r45gj Stopping container istio-proxy 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5 Created pod: payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-477f6b Scaled up replica set payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5 from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-477f6b-58f667c489 Error creating: pods "payload-processing-e2e-shared-a-w6-477f6b-58f667c489-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b: serviceaccount "payload-processing-e2e-shared-a-w6-477f6b" not found 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5 Error creating: pods "payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b: serviceaccount "payload-processing-e2e-shared-a-w6-477f6b" not found 47m Normal AddedLabel gateway/e2e-api-a-3571f8 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-3571f8 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-477f6b Scaled up replica set payload-processing-e2e-shared-a-w6-477f6b-58f667c489 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-477f6b-58f667c489 Created pod: payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf 47m Normal SuccessfulCreate replicaset/e2e-api-a-3571f8-openshift-default-5f85d888fc Created pod: e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v 47m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Add eth0 [10.134.0.86/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/e2e-api-a-3571f8-openshift-default Scaled up replica set e2e-api-a-3571f8-openshift-default-5f85d888fc from 0 to 1 47m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Add eth0 [10.134.0.85/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Container created 47m Normal AddedInterface pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Add eth0 [10.134.0.87/23] from ovn-kubernetes 47m Normal Started pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Container started 47m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Container started 47m Normal Created pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Container created 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9 Error creating: pods "payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-815776: serviceaccount "payload-processing-e2e-shared-a-w4-815776" not found 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9 Created pod: payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9 Error creating: pods "payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-815776: serviceaccount "payload-processing-e2e-shared-a-w4-815776" not found 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w4-815776 Scaled up replica set payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w4-815776 Scaled up replica set payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9 Created pod: payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt 47m Normal AddedInterface pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Add eth0 [10.133.0.168/23] from ovn-kubernetes 47m Normal Started pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Container started 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-3571f8 Scaled up replica set payload-processing-e2e-api-a-3571f8-765764b5dc from 0 to 1 47m Normal Created pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Container created 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-3571f8 Scaled up replica set payload-pre-processing-e2e-api-a-3571f8-c7855d6bc from 0 to 1 47m Normal Pulled pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Warning FailedCreate replicaset/payload-processing-e2e-api-a-3571f8-765764b5dc Error creating: pods "payload-processing-e2e-api-a-3571f8-765764b5dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3571f8: serviceaccount "payload-processing-e2e-api-a-3571f8" not found 47m Normal Created pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Container created 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc Created pod: payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk 47m Normal Started pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Container started 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc Error creating: pods "payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-3571f8: serviceaccount "payload-processing-e2e-api-a-3571f8" not found 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Add eth0 [10.133.0.166/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Add eth0 [10.133.0.169/23] from ovn-kubernetes 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-3571f8-765764b5dc Created pod: payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc 47m Normal Created pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Container created 47m Normal Pulled pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Add eth0 [10.133.0.170/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Container started 47m Normal Started pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Container started 47m Normal Created pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Container created 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 47m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled up replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 1 to 4 47m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Container started 47m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Add eth0 [10.134.0.88/23] from ovn-kubernetes 47m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Container created 47m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Container started 47m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Container created 47m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Add eth0 [10.132.0.102/23] from ovn-kubernetes 47m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Container started 47m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Container created 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 47m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Add eth0 [10.134.0.89/23] from ovn-kubernetes 47m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Container created 47m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Container started 47m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Startup probe error: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Startup probe failed: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Startup probe error: Get "http://10.134.0.89:15021/healthz/ready": dial tcp 10.134.0.89:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Startup probe failed: Get "http://10.134.0.89:15021/healthz/ready": dial tcp 10.134.0.89:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Startup probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Startup probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Startup probe error: Get "http://10.133.0.158:15021/healthz/ready": dial tcp 10.133.0.158:15021: connect: connection refused... 47m Normal AddedLabel gateway/e2e-api-a-d4ccc5 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-d4ccc5 47m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Startup probe failed: Get "http://10.133.0.158:15021/healthz/ready": dial tcp 10.133.0.158:15021: connect: connection refused 47m Normal SuccessfulCreate replicaset/e2e-api-a-d4ccc5-openshift-default-646f9fd445 Created pod: e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq 47m Normal ScalingReplicaSet deployment/e2e-api-a-d4ccc5-openshift-default Scaled up replica set e2e-api-a-d4ccc5-openshift-default-646f9fd445 from 0 to 1 47m Normal AddedInterface pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Add eth0 [10.133.0.174/23] from ovn-kubernetes 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f Error creating: pods "payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d4ccc5: serviceaccount "payload-processing-e2e-api-a-d4ccc5" not found 47m Warning FailedCreate replicaset/payload-processing-e2e-api-a-d4ccc5-66cf5d675f Error creating: pods "payload-processing-e2e-api-a-d4ccc5-66cf5d675f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d4ccc5: serviceaccount "payload-processing-e2e-api-a-d4ccc5" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-d4ccc5-66cf5d675f Created pod: payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-d4ccc5 Scaled up replica set payload-processing-e2e-api-a-d4ccc5-66cf5d675f from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f Created pod: payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-d4ccc5 Scaled up replica set payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f from 0 to 1 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Started pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Container started 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Add eth0 [10.133.0.176/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Add eth0 [10.133.0.175/23] from ovn-kubernetes 47m Normal Created pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Container created 47m Normal Created pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Container created 47m Normal Started pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Container started 47m Normal Pulled pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning Unhealthy pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Readiness probe failed: Get "http://10.133.0.163:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Warning ProbeError pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Readiness probe error: Get "http://10.133.0.163:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 46m Normal Started pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Container started 46m Normal Pulled pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Container started 46m Normal Pulled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Container created 46m Normal Created pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Container created 46m Normal AddedLabel gateway/e2e-shared-b-w6-f22a52 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-f22a52 46m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f Created pod: e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv 46m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-f22a52-openshift-default Scaled up replica set e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f from 0 to 1 46m Normal AddedInterface pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Add eth0 [10.133.0.178/23] from ovn-kubernetes 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-1b2243ef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-1b2243ef-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Created pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 46m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-cc33d3-openshift-default Scaled up replica set e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 from 0 to 1 46m Normal AddedLabel gateway/e2e-shared-b-w4-cc33d3 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w4-cc33d3 46m Warning Unhealthy pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Startup probe failed: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused 46m Normal AddedInterface pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Add eth0 [10.134.0.91/23] from ovn-kubernetes 46m Warning ProbeError pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Startup probe error: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused... 46m Normal ScalingReplicaSet deployment/e2e-api-b-4fc75f-openshift-default Scaled up replica set e2e-api-b-4fc75f-openshift-default-5867848d5c from 0 to 1 46m Normal AddedLabel gateway/e2e-api-b-4fc75f Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-4fc75f 46m Normal SuccessfulCreate replicaset/e2e-api-b-4fc75f-openshift-default-5867848d5c Created pod: e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms 46m Warning ProbeError pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Startup probe error: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Startup probe failed: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff Error creating: pods "payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52: serviceaccount "payload-processing-e2e-shared-b-w6-f22a52" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff Created pod: payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-f22a52 Scaled up replica set payload-processing-e2e-shared-b-w6-f22a52-895f8cfff from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-f22a52 Scaled up replica set payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf from 0 to 1 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf Error creating: pods "payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52: serviceaccount "payload-processing-e2e-shared-b-w6-f22a52" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf Created pod: payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl 46m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Container started 46m Normal AddedInterface pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Add eth0 [10.133.0.179/23] from ovn-kubernetes 46m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Container created 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Add eth0 [10.134.0.92/23] from ovn-kubernetes 46m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Container created 46m Normal Started pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Container started 46m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Add eth0 [10.133.0.180/23] from ovn-kubernetes 46m Normal Pulled pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w4-cc33d3 Scaled up replica set payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc Error creating: pods "payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3: serviceaccount "payload-processing-e2e-shared-b-w4-cc33d3" not found 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w4-cc33d3 Scaled up replica set payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f Created pod: payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc Created pod: payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f Error creating: pods "payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3: serviceaccount "payload-processing-e2e-shared-b-w4-cc33d3" not found 46m Normal Created pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Container created 46m Normal AddedInterface pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Add eth0 [10.133.0.182/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Container started 46m Normal Created pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Container created 46m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Started pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Container started 46m Normal Pulled pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Add eth0 [10.133.0.181/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-4fc75f-847c6dd8b7 Created pod: payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-4fc75f Scaled up replica set payload-processing-e2e-api-b-4fc75f-847c6dd8b7 from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-4fc75f Scaled up replica set payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-api-b-4fc75f-847c6dd8b7 Error creating: pods "payload-processing-e2e-api-b-4fc75f-847c6dd8b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-4fc75f: serviceaccount "payload-processing-e2e-api-b-4fc75f" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c Error creating: pods "payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-4fc75f: serviceaccount "payload-processing-e2e-api-b-4fc75f" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c Created pod: payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Add eth0 [10.133.0.183/23] from ovn-kubernetes 46m Normal Pulled pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Pulled pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Container created 46m Normal AddedInterface pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Add eth0 [10.134.0.93/23] from ovn-kubernetes 46m Normal Created pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Container created 46m Normal Started pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Container started 46m Normal Started pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Container started 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-1b2243ef-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-1b2243ef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal AddedLabel gateway/e2e-api-b-b202a0 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-b202a0 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-477f6b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-477f6b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal SuccessfulCreate replicaset/e2e-api-b-b202a0-openshift-default-5756cfdf4f Created pod: e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw 46m Normal ScalingReplicaSet deployment/e2e-api-b-b202a0-openshift-default Scaled up replica set e2e-api-b-b202a0-openshift-default-5756cfdf4f from 0 to 1 46m Normal AddedInterface pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Add eth0 [10.134.0.94/23] from ovn-kubernetes 46m Warning Unhealthy pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Readiness probe failed: Get "http://10.133.0.174:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Warning ProbeError pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Readiness probe error: Get "http://10.133.0.174:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 46m Normal AddedLabel gateway/e2e-mt-a3efab09 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-a3efab09 46m Normal SuccessfulCreate replicaset/e2e-mt-a3efab09-openshift-default-789b59cc4f Created pod: e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb 46m Normal ScalingReplicaSet deployment/e2e-mt-a3efab09-openshift-default Scaled up replica set e2e-mt-a3efab09-openshift-default-789b59cc4f from 0 to 1 46m Normal AddedInterface pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Add eth0 [10.134.0.95/23] from ovn-kubernetes 46m Normal Pulled pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Container created 46m Normal Started pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Container started 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-815776-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-1b2243ef-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 46m Normal ScalingReplicaSet deployment/e2e-mt-1b2243ef-openshift-default Scaled up replica set e2e-mt-1b2243ef-openshift-default-88dfb6485 from 1 to 2 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-815776-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal SuccessfulCreate replicaset/e2e-mt-1b2243ef-openshift-default-88dfb6485 Created pod: e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-b202a0 Scaled up replica set payload-processing-e2e-api-b-b202a0-579d586c5f from 0 to 1 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal Created pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Container created 46m Normal Started pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Container started 46m Normal AddedInterface pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Add eth0 [10.132.0.107/23] from ovn-kubernetes 46m Normal Created pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Container created 46m Normal Pulled pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Container started 46m Normal Pulled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Warning FailedCreate replicaset/payload-processing-e2e-api-b-b202a0-579d586c5f Error creating: pods "payload-processing-e2e-api-b-b202a0-579d586c5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-b202a0: serviceaccount "payload-processing-e2e-api-b-b202a0" not found 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-b202a0 Scaled up replica set payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7 from 0 to 1 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7 Error creating: pods "payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-b202a0: serviceaccount "payload-processing-e2e-api-b-b202a0" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7 Created pod: payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-b202a0-579d586c5f Created pod: payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Add eth0 [10.133.0.184/23] from ovn-kubernetes 46m Normal Started pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Container started 46m Normal Pulled pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Add eth0 [10.134.0.96/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Container started 46m Normal Created pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Container created 46m Normal Created pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Container created 46m Normal Pulled pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-a3efab09-6f7cc965b5 Created pod: payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 46m Warning Unhealthy pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Startup probe failed: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-a3efab09 Scaled up replica set payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5 from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-a3efab09 Scaled up replica set payload-processing-e2e-mt-a3efab09-6f7cc965b5 from 0 to 1 46m Warning ProbeError pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Startup probe error: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused... 46m Warning FailedCreate replicaset/payload-processing-e2e-mt-a3efab09-6f7cc965b5 Error creating: pods "payload-processing-e2e-mt-a3efab09-6f7cc965b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-a3efab09: serviceaccount "payload-processing-e2e-mt-a3efab09" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5 Error creating: pods "payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-a3efab09: serviceaccount "payload-processing-e2e-mt-a3efab09" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5 Created pod: payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 46m Warning Unhealthy pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Startup probe failed: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused 46m Normal AddedLabel gateway/e2e-inf-a-862dc2 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-862dc2 46m Warning ProbeError pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Startup probe error: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused... 46m Normal ScalingReplicaSet deployment/e2e-inf-a-862dc2-openshift-default Scaled up replica set e2e-inf-a-862dc2-openshift-default-66cbdd88d9 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-inf-a-862dc2-openshift-default-66cbdd88d9 Created pod: e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr 46m Warning ProbeError pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Startup probe error: Get "http://10.133.0.174:15021/healthz/ready": dial tcp 10.133.0.174:15021: connect: connection refused... 46m Normal AddedInterface pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Add eth0 [10.133.0.186/23] from ovn-kubernetes 46m Warning Unhealthy pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Startup probe failed: Get "http://10.133.0.174:15021/healthz/ready": dial tcp 10.133.0.174:15021: connect: connection refused 46m Normal AddedInterface pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Add eth0 [10.133.0.187/23] from ovn-kubernetes 46m Normal AddedInterface pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Add eth0 [10.133.0.185/23] from ovn-kubernetes 46m Normal Started pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Container started 46m Normal Created pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Container created 46m Normal Pulled pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Pulled pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Container created 46m Normal Started pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Container started 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-862dc2 Scaled up replica set payload-processing-e2e-inf-a-862dc2-757889cb9f from 0 to 1 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c Error creating: pods "payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-862dc2: serviceaccount "payload-processing-e2e-inf-a-862dc2" not found 45m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-862dc2-757889cb9f Error creating: pods "payload-processing-e2e-inf-a-862dc2-757889cb9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-862dc2: serviceaccount "payload-processing-e2e-inf-a-862dc2" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-862dc2-757889cb9f Created pod: payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-862dc2 Scaled up replica set payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c Created pod: payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr 45m Normal Created pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Container created 45m Normal Pulled pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Container started 45m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Add eth0 [10.134.0.97/23] from ovn-kubernetes 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d4ccc5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d4ccc5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 4; reason: cpu resource utilization (percentage of request) above target 45m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-6rrhv 45m Normal Killing pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-d8vdz Stopping container istio-proxy 45m Normal SuccessfulCreate replicaset/e2e-mt-1b2243ef-openshift-default-88dfb6485 Created pod: e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp 45m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 4 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-815776-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-815776-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 45m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Container started 45m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Readiness probe error: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Readiness probe failed: Get "http://10.134.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Add eth0 [10.132.0.109/23] from ovn-kubernetes 45m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Container created 45m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Readiness probe error: HTTP probe failed with statuscode: 503... 45m Normal Created pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Container created 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d4ccc5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d4ccc5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal Pulled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Container started 45m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused 45m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused... 45m Normal Killing pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Stopping container payload-pre-processing 45m Normal Killing pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Stopping container payload-processing 45m Warning Unhealthy pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled down replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 10 to 8 45m Warning Unhealthy pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 8; reason: All metrics below target 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 45m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled down replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 10 to 7 45m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz 45m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 7; reason: All metrics below target 44m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Readiness probe error: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf Readiness probe failed: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Readiness probe error: Get "http://10.133.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Readiness probe failed: Get "http://10.134.0.94:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal Started pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Container started 44m Normal Created pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Container created 44m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Readiness probe error: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning ProbeError pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Readiness probe error: Get "http://10.134.0.94:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd Readiness probe failed: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf Readiness probe failed: Get "http://10.133.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal AddedInterface pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Add eth0 [10.134.0.98/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Container created 44m Normal AddedInterface pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Add eth0 [10.133.0.194/23] from ovn-kubernetes 44m Normal Pulled pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Container started 44m Normal Started pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Container started 44m Normal Created pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Container created 44m Normal Pulled pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal ScalingReplicaSet deployment/e2e-api-b-4fc75f-openshift-default Scaled up replica set e2e-api-b-4fc75f-openshift-default-5867848d5c from 1 to 3 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-api-b-4fc75f-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 44m Normal SuccessfulCreate replicaset/e2e-api-b-4fc75f-openshift-default-5867848d5c Created pod: e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp 44m Normal SuccessfulCreate replicaset/e2e-api-b-4fc75f-openshift-default-5867848d5c Created pod: e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z 44m Warning ProbeError pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Startup probe error: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused... 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 7; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn 44m Warning Unhealthy pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Startup probe failed: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused 44m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled down replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 8 to 7 44m Normal Started pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Container started 44m Warning Unhealthy pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Readiness probe failed: Get "http://10.133.0.179:15021/healthz/ready": read tcp 10.133.0.2:58600->10.133.0.179:15021: read: connection reset by peer 44m Warning Unhealthy pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Startup probe failed: Get "http://10.134.0.94:15021/healthz/ready": dial tcp 10.134.0.94:15021: connect: connection refused 44m Warning ProbeError pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Startup probe error: Get "http://10.134.0.94:15021/healthz/ready": dial tcp 10.134.0.94:15021: connect: connection refused... 44m Normal Created pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Container created 44m Normal Pulled pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning ProbeError pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Readiness probe error: Get "http://10.133.0.179:15021/healthz/ready": read tcp 10.133.0.2:58600->10.133.0.179:15021: read: connection reset by peer... 44m Normal Pulled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Container created 44m Normal Started pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Container started 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-b202a0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-b202a0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal AddedInterface pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Add eth0 [10.134.0.99/23] from ovn-kubernetes 44m Normal Created pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Container created 44m Normal Pulled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Container started 44m Warning Unhealthy pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Startup probe failed: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused 44m Warning ProbeError pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Startup probe error: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Startup probe failed: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused 44m Warning ProbeError pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Startup probe error: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Startup probe failed: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused 44m Warning ProbeError pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Startup probe error: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused... 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-4fc75f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-4fc75f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-862dc2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-862dc2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-b202a0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-b202a0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled down replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 10 to 6 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 44m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled down replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 10 to 7 44m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh 44m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Stopping container istio-proxy 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 7; reason: All metrics below target 44m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t 44m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh Stopping container istio-proxy 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 6; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp 44m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st 44m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-862dc2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-862dc2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal Started pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Container started 44m Normal Created pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Container created 44m Normal Pulled pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-a3efab09-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-a3efab09-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal AddedInterface pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Add eth0 [10.133.0.197/23] from ovn-kubernetes 44m Normal Pulled pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Container created 44m Normal Started pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Container started 44m Normal AddedLabel gateway/e2e-inf-b-de7242 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-de7242 44m Normal ScalingReplicaSet deployment/e2e-inf-b-de7242-openshift-default Scaled up replica set e2e-inf-b-de7242-openshift-default-7cf5999677 from 0 to 1 44m Normal SuccessfulCreate replicaset/e2e-inf-b-de7242-openshift-default-7cf5999677 Created pod: e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z 44m Normal AddedInterface pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Add eth0 [10.133.0.201/23] from ovn-kubernetes 44m Warning ProbeError pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Startup probe error: Get "http://10.134.0.95:15021/healthz/ready": dial tcp 10.134.0.95:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Startup probe failed: Get "http://10.134.0.95:15021/healthz/ready": dial tcp 10.134.0.95:15021: connect: connection refused 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-4fc75f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-4fc75f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning ProbeError pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Startup probe error: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Startup probe failed: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf Error creating: pods "payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1b2243ef: serviceaccount "payload-processing-e2e-mt-1b2243ef" not found 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-1b2243ef-7955fbbb49 Created pod: payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z 44m Warning FailedCreate replicaset/payload-processing-e2e-mt-1b2243ef-7955fbbb49 Error creating: pods "payload-processing-e2e-mt-1b2243ef-7955fbbb49-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1b2243ef: serviceaccount "payload-processing-e2e-mt-1b2243ef" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-1b2243ef Scaled up replica set payload-processing-e2e-mt-1b2243ef-7955fbbb49 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf Created pod: payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-1b2243ef Scaled up replica set payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf from 0 to 1 44m Normal Created pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Container created 44m Normal AddedInterface pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Add eth0 [10.133.0.202/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Container started 44m Normal AddedInterface pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Add eth0 [10.133.0.203/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c Error creating: pods "payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-de7242: serviceaccount "payload-processing-e2e-inf-b-de7242" not found 44m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-de7242-777f988ccf Error creating: pods "payload-processing-e2e-inf-b-de7242-777f988ccf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-de7242: serviceaccount "payload-processing-e2e-inf-b-de7242" not found 44m Normal Created pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Container created 44m Normal Started pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Container started 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-de7242-777f988ccf Created pod: payload-processing-e2e-inf-b-de7242-777f988ccf-879xx 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-de7242 Scaled up replica set payload-processing-e2e-inf-b-de7242-777f988ccf from 0 to 1 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-de7242 Scaled up replica set payload-pre-processing-e2e-inf-b-de7242-5f4bd94c from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c Created pod: payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j 44m Normal Pulled pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Container started 44m Normal Created pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Container created 44m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Add eth0 [10.134.0.100/23] from ovn-kubernetes 44m Normal Started pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Container started 44m Normal Created pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Container created 44m Normal Pulled pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Add eth0 [10.133.0.204/23] from ovn-kubernetes 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z 43m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled down replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 10 to 7 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 7; reason: All metrics below target 43m Normal Killing pod/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b Stopping container payload-pre-processing 43m Normal Killing pod/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5 Stopping container payload-processing 43m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z Stopping container istio-proxy 43m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Readiness probe error: HTTP probe failed with statuscode: 503... 43m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Stopping container istio-proxy 43m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn Readiness probe error: HTTP probe failed with statuscode: 503... 43m Normal Killing pod/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk Stopping container payload-pre-processing 43m Normal Killing pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Stopping container payload-processing 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-a3efab09-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-a3efab09-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning Unhealthy pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Readiness probe failed: Get "http://10.133.0.178:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Readiness probe error: Get "http://10.133.0.178:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Pulled pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Started pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Container started 43m Normal Created pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Container created 43m Warning Unhealthy pod/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc Readiness probe failed: timeout: failed to connect service "10.133.0.170:50052" within 1s: context deadline exceeded 43m Warning Unhealthy pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Readiness probe failed: Get "http://10.134.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Readiness probe error: Get "http://10.134.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning ProbeError pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Startup probe error: Get "http://10.133.0.178:15021/healthz/ready": dial tcp 10.133.0.178:15021: connect: connection refused... 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 8; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled down replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 10 to 8 43m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm Stopping container istio-proxy 43m Warning Unhealthy pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Startup probe failed: Get "http://10.133.0.178:15021/healthz/ready": dial tcp 10.133.0.178:15021: connect: connection refused 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-vfwgs 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-hfblm 43m Normal Started pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Container started 43m Normal Created pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Container created 43m Normal Pulled pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning ProbeError pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Startup probe error: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Startup probe failed: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused 43m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled down replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 7 to 6 43m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f Stopping container istio-proxy 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 7; reason: All metrics below target 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 6; reason: All metrics below target 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 6; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled down replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 8 to 6 43m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 43m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl 43m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl 43m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp 43m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled down replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 10 to 7 43m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp Stopping container istio-proxy 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Readiness probe error: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Readiness probe error: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Readiness probe failed: Get "http://10.132.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Readiness probe failed: Get "http://10.132.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Readiness probe error: Get "http://10.134.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Readiness probe failed: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Readiness probe failed: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Readiness probe error: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Readiness probe failed: Get "http://10.134.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds Readiness probe error: Get "http://10.133.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-3571f8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-3571f8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-3571f8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-3571f8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Normal Killing pod/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6 Stopping container payload-processing 42m Normal Killing pod/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5 Stopping container payload-pre-processing 42m Normal Killing pod/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m Stopping container payload-pre-processing 42m Normal Killing pod/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z Stopping container payload-processing 42m Normal Killing pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Stopping container istio-proxy 42m Warning Unhealthy pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Readiness probe failed: HTTP probe failed with statuscode: 503 42m Warning ProbeError pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq Readiness probe error: HTTP probe failed with statuscode: 503... 42m Normal Killing pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Stopping container istio-proxy 42m Normal Killing pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb Stopping container istio-proxy 42m Normal ScalingReplicaSet deployment/e2e-api-a-84dc9d-openshift-default Scaled up replica set e2e-api-a-84dc9d-openshift-default-54749c6469 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-api-a-84dc9d-openshift-default-54749c6469 Created pod: e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 42m Normal AddedLabel gateway/e2e-api-a-84dc9d Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-84dc9d 42m Normal AddedInterface pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Add eth0 [10.133.0.210/23] from ovn-kubernetes 42m Normal SuccessfulDelete replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Deleted pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx 42m Warning ProbeError pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Readiness probe error: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused... 42m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 Stopping container istio-proxy 42m Normal SuccessfulDelete replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Deleted pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 42m Warning Unhealthy pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v Readiness probe failed: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused 42m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx Stopping container istio-proxy 42m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled down replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 4 to 2 42m Normal Killing pod/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2 Stopping container payload-processing 42m Normal Killing pod/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn Stopping container payload-pre-processing 42m Normal Created pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Container created 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-84dc9d Scaled up replica set payload-processing-e2e-api-a-84dc9d-5657dc7c79 from 0 to 1 42m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Readiness probe failed: Get "http://10.133.0.201:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-84dc9d Scaled up replica set payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5 Created pod: payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5 Error creating: pods "payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-84dc9d: serviceaccount "payload-processing-e2e-api-a-84dc9d" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-84dc9d-5657dc7c79 Created pod: payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 42m Normal Pulled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Warning FailedCreate replicaset/payload-processing-e2e-api-a-84dc9d-5657dc7c79 Error creating: pods "payload-processing-e2e-api-a-84dc9d-5657dc7c79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-84dc9d: serviceaccount "payload-processing-e2e-api-a-84dc9d" not found 42m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Readiness probe error: Get "http://10.133.0.201:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Normal Started pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Container started 42m Normal Started pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Container started 42m Normal Created pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Container created 42m Normal Created pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Container created 42m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Add eth0 [10.133.0.212/23] from ovn-kubernetes 42m Normal Started pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Container started 42m Normal Pulled pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Pulled pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Add eth0 [10.133.0.211/23] from ovn-kubernetes 42m Normal Killing pod/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd Stopping container payload-processing 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 6; reason: All metrics below target 42m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default Scaled down replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 7 to 6 42m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc 42m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9fgqc Stopping container istio-proxy 42m Normal Killing pod/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm Stopping container payload-pre-processing 42m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd 42m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd Stopping container istio-proxy 42m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled down replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 7 to 6 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 6; reason: All metrics below target 42m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Startup probe error: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Startup probe failed: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal SuccessfulCreate replicaset/e2e-inf-a-f4e35c-openshift-default-5b758f58b7 Created pod: e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs 41m Normal ScalingReplicaSet deployment/e2e-inf-a-f4e35c-openshift-default Scaled up replica set e2e-inf-a-f4e35c-openshift-default-5b758f58b7 from 0 to 1 41m Normal AddedLabel gateway/e2e-inf-a-f4e35c Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-f4e35c 41m Normal AddedInterface pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Add eth0 [10.133.0.216/23] from ovn-kubernetes 41m Normal Killing pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Stopping container istio-proxy 41m Warning ProbeError pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Readiness probe error: HTTP probe failed with statuscode: 503... 41m Warning Unhealthy pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp Readiness probe failed: HTTP probe failed with statuscode: 503 41m Normal Killing pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-f4e35c Scaled up replica set payload-processing-e2e-inf-a-f4e35c-66ff966c5c from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-f4e35c Scaled up replica set payload-pre-processing-e2e-inf-a-f4e35c-c74d84667 from 0 to 1 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-f4e35c-66ff966c5c Error creating: pods "payload-processing-e2e-inf-a-f4e35c-66ff966c5c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f4e35c: serviceaccount "payload-processing-e2e-inf-a-f4e35c" not found 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Add eth0 [10.133.0.218/23] from ovn-kubernetes 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667 Created pod: payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-f4e35c-66ff966c5c Created pod: payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667 Error creating: pods "payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f4e35c: serviceaccount "payload-processing-e2e-inf-a-f4e35c" not found 41m Normal Pulled pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Add eth0 [10.133.0.219/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Container started 41m Normal Created pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Container created 41m Normal Started pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Container started 41m Normal Created pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Container created 41m Normal Killing pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw Stopping container istio-proxy 41m Normal AddedLabel gateway/e2e-inf-a-c1d09f Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-c1d09f 41m Normal SuccessfulCreate replicaset/e2e-inf-a-c1d09f-openshift-default-67b97686fc Created pod: e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 41m Normal ScalingReplicaSet deployment/e2e-inf-a-c1d09f-openshift-default Scaled up replica set e2e-inf-a-c1d09f-openshift-default-67b97686fc from 0 to 1 41m Normal ScalingReplicaSet deployment/e2e-inf-a-aee070-openshift-default Scaled up replica set e2e-inf-a-aee070-openshift-default-5b997d785f from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-a-aee070-openshift-default-5b997d785f Created pod: e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 41m Normal AddedLabel gateway/e2e-inf-a-aee070 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-aee070 41m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Readiness probe failed: Get "http://10.133.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Readiness probe error: Get "http://10.133.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Normal AddedInterface pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Add eth0 [10.133.0.228/23] from ovn-kubernetes 41m Normal AddedInterface pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Add eth0 [10.133.0.227/23] from ovn-kubernetes 41m Normal AddedLabel gateway/e2e-api-b-f78ca5 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-f78ca5 41m Normal ScalingReplicaSet deployment/e2e-api-b-f78ca5-openshift-default Scaled up replica set e2e-api-b-f78ca5-openshift-default-d6b5dd7dc from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc Created pod: e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j 41m Normal AddedInterface pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Add eth0 [10.133.0.229/23] from ovn-kubernetes 41m Normal Killing pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Stopping container istio-proxy 41m Normal Killing pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z Stopping container istio-proxy 41m Normal Killing pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/e2e-inf-a-387b16-openshift-default Scaled up replica set e2e-inf-a-387b16-openshift-default-5bb4745897 from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr 41m Normal AddedLabel gateway/e2e-inf-a-387b16 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-387b16 41m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Add eth0 [10.133.0.230/23] from ovn-kubernetes 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5 Error creating: pods "payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-c1d09f: serviceaccount "payload-processing-e2e-inf-a-c1d09f" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5 Created pod: payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-c1d09f Scaled up replica set payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5 from 0 to 1 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-c1d09f-7f6b89c47 Error creating: pods "payload-processing-e2e-inf-a-c1d09f-7f6b89c47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-c1d09f: serviceaccount "payload-processing-e2e-inf-a-c1d09f" not found 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-c1d09f-7f6b89c47 Created pod: payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-c1d09f Scaled up replica set payload-processing-e2e-inf-a-c1d09f-7f6b89c47 from 0 to 1 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Add eth0 [10.133.0.231/23] from ovn-kubernetes 41m Normal Created pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Container created 41m Normal SuccessfulCreate replicaset/e2e-inf-b-de7242-openshift-default-7cf5999677 Created pod: e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 41m Normal Pulled pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Add eth0 [10.134.0.105/23] from ovn-kubernetes 41m Normal Started pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Container started 41m Normal ScalingReplicaSet deployment/e2e-inf-b-de7242-openshift-default Scaled up replica set e2e-inf-b-de7242-openshift-default-7cf5999677 from 1 to 2 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 41m Normal Created pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Container created 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc Created pod: payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh 41m Normal Pulled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Container started 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-aee070 Scaled up replica set payload-processing-e2e-inf-a-aee070-65899ddbd9 from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc Error creating: pods "payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-aee070: serviceaccount "payload-processing-e2e-inf-a-aee070" not found 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-aee070-65899ddbd9 Created pod: payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg 41m Normal Created pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Container created 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-aee070-65899ddbd9 Error creating: pods "payload-processing-e2e-inf-a-aee070-65899ddbd9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-aee070: serviceaccount "payload-processing-e2e-inf-a-aee070" not found 41m Normal AddedInterface pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Add eth0 [10.134.0.106/23] from ovn-kubernetes 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-aee070 Scaled up replica set payload-pre-processing-e2e-inf-a-aee070-ff89c7dc from 0 to 1 41m Warning Unhealthy pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Readiness probe failed: Get "http://10.133.0.179:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Warning Unhealthy pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Readiness probe failed: Get "http://10.133.0.210:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Warning ProbeError pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Readiness probe error: Get "http://10.133.0.210:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Warning ProbeError pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms Readiness probe error: Get "http://10.133.0.179:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-f78ca5-679f7559f Error creating: pods "payload-pre-processing-e2e-api-b-f78ca5-679f7559f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-f78ca5: serviceaccount "payload-processing-e2e-api-b-f78ca5" not found 41m Normal Started pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Container started 41m Warning FailedCreate replicaset/payload-processing-e2e-api-b-f78ca5-78696c64f Error creating: pods "payload-processing-e2e-api-b-f78ca5-78696c64f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-f78ca5: serviceaccount "payload-processing-e2e-api-b-f78ca5" not found 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-f78ca5 Scaled up replica set payload-pre-processing-e2e-api-b-f78ca5-679f7559f from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-f78ca5 Scaled up replica set payload-processing-e2e-api-b-f78ca5-78696c64f from 0 to 1 41m Normal Pulled pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Add eth0 [10.133.0.233/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Container created 41m Normal Started pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Container started 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-f78ca5-78696c64f Created pod: payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-f78ca5-679f7559f Created pod: payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Add eth0 [10.133.0.232/23] from ovn-kubernetes 41m Normal Created pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Container created 41m Normal Started pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Container started 41m Normal Started pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Container started 41m Normal Started pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Container started 41m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Add eth0 [10.133.0.234/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Container created 41m Normal Created pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Container created 41m Normal Pulled pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Add eth0 [10.133.0.235/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Container created 41m Normal Started pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Container started 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-387b16-569d556879 Error creating: pods "payload-processing-e2e-inf-a-387b16-569d556879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-387b16: serviceaccount "payload-processing-e2e-inf-a-387b16" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-387b16 Scaled up replica set payload-processing-e2e-inf-a-387b16-569d556879 from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-387b16 Scaled up replica set payload-pre-processing-e2e-inf-a-387b16-75454f7c9f from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f Created pod: payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f Error creating: pods "payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-387b16: serviceaccount "payload-processing-e2e-inf-a-387b16" not found 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-387b16-569d556879 Created pod: payload-processing-e2e-inf-a-387b16-569d556879-gmrbp 41m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl 41m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-ff5kl Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled down replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 7 to 6 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 6; reason: All metrics below target 41m Normal Started pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Container started 41m Normal Created pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Container created 41m Normal Started pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Container started 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Add eth0 [10.133.0.237/23] from ovn-kubernetes 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Add eth0 [10.133.0.236/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Container created 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Startup probe failed: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused 41m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Startup probe error: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Startup probe failed: Get "http://10.133.0.210:15021/healthz/ready": dial tcp 10.133.0.210:15021: connect: connection refused 41m Warning ProbeError pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Startup probe error: Get "http://10.133.0.210:15021/healthz/ready": dial tcp 10.133.0.210:15021: connect: connection refused... 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-84dc9d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-84dc9d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-84dc9d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-84dc9d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn 40m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-tg9fl 40m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled down replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 6 to 5 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 5; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn Stopping container istio-proxy 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default New size: 5; reason: All metrics below target 40m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled down replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 6 to 5 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 5; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 5; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 6; reason: All metrics below target 40m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled down replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 7 to 6 40m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-w4wbg 40m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled down replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 6 to 5 40m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Stopping container istio-proxy 40m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Readiness probe error: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 40m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz Readiness probe failed: Get "http://10.134.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 40m Normal Started pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Container started 40m Normal Created pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Container created 40m Normal Pulled pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal ScalingReplicaSet deployment/e2e-inf-a-7df9fa-openshift-default Scaled up replica set e2e-inf-a-7df9fa-openshift-default-b67f7d48f from 0 to 1 40m Normal AddedLabel gateway/e2e-inf-a-7df9fa Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-7df9fa 40m Normal SuccessfulCreate replicaset/e2e-inf-a-7df9fa-openshift-default-b67f7d48f Created pod: e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz 40m Normal AddedInterface pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Add eth0 [10.134.0.110/23] from ovn-kubernetes 40m Warning Unhealthy pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Startup probe failed: Get "http://10.133.0.216:15021/healthz/ready": dial tcp 10.133.0.216:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Startup probe error: Get "http://10.133.0.216:15021/healthz/ready": dial tcp 10.133.0.216:15021: connect: connection refused... 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-7df9fa Scaled up replica set payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c from 0 to 1 40m Normal AddedInterface pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Add eth0 [10.134.0.112/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-7df9fa Scaled up replica set payload-processing-e2e-inf-a-7df9fa-cc7758555 from 0 to 1 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c Error creating: pods "payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-7df9fa: serviceaccount "payload-processing-e2e-inf-a-7df9fa" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c Created pod: payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-7df9fa-cc7758555 Created pod: payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-7df9fa-cc7758555 Error creating: pods "payload-processing-e2e-inf-a-7df9fa-cc7758555-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-7df9fa: serviceaccount "payload-processing-e2e-inf-a-7df9fa" not found 40m Normal Pulled pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Add eth0 [10.134.0.111/23] from ovn-kubernetes 40m Normal Started pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Container started 40m Normal Created pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Container created 40m Normal Created pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Container created 40m Normal Pulled pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Started pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Container started 40m Normal SuccessfulCreate replicaset/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff Created pod: e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n 40m Normal ScalingReplicaSet deployment/e2e-inf-b-3cc570-openshift-default Scaled up replica set e2e-inf-b-3cc570-openshift-default-7f6b9c66ff from 0 to 1 40m Normal AddedLabel gateway/e2e-inf-b-3cc570 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-3cc570 40m Normal AddedInterface pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Add eth0 [10.134.0.113/23] from ovn-kubernetes 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-f4e35c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-f4e35c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-3cc570-7cc44d69c Error creating: pods "payload-processing-e2e-inf-b-3cc570-7cc44d69c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-3cc570: serviceaccount "payload-processing-e2e-inf-b-3cc570" not found 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-3cc570-7cc44d69c Created pod: payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-3cc570 Scaled up replica set payload-processing-e2e-inf-b-3cc570-7cc44d69c from 0 to 1 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc Error creating: pods "payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-3cc570: serviceaccount "payload-processing-e2e-inf-b-3cc570" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc Created pod: payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-3cc570 Scaled up replica set payload-pre-processing-e2e-inf-b-3cc570-d49b598dc from 0 to 1 40m Normal Started pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Container started 40m Normal Pulled pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Container created 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Add eth0 [10.134.0.114/23] from ovn-kubernetes 40m Normal Pulled pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Container created 40m Normal AddedInterface pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Add eth0 [10.133.0.240/23] from ovn-kubernetes 40m Normal Started pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Container started 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-862dc2-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal AddedInterface pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Add eth0 [10.134.0.115/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal ScalingReplicaSet deployment/e2e-inf-a-862dc2-openshift-default Scaled up replica set e2e-inf-a-862dc2-openshift-default-66cbdd88d9 from 1 to 2 40m Normal SuccessfulCreate replicaset/e2e-inf-a-862dc2-openshift-default-66cbdd88d9 Created pod: e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs 40m Normal Created pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Container created 40m Normal Started pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Container started 39m Warning ProbeError pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Startup probe error: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Startup probe failed: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-f4e35c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-f4e35c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal SuccessfulCreate replicaset/e2e-inf-b-62316c-openshift-default-96996f9c9 Created pod: e2e-inf-b-62316c-openshift-default-96996f9c9-q792b 39m Normal AddedLabel gateway/e2e-inf-b-62316c Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-62316c 39m Normal AddedLabel gateway/e2e-inf-b-c54211 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-c54211 39m Normal ScalingReplicaSet deployment/e2e-inf-b-62316c-openshift-default Scaled up replica set e2e-inf-b-62316c-openshift-default-96996f9c9 from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-inf-b-c54211-openshift-default-767566b477 Created pod: e2e-inf-b-c54211-openshift-default-767566b477-nvw4t 39m Normal ScalingReplicaSet deployment/e2e-inf-b-c54211-openshift-default Scaled up replica set e2e-inf-b-c54211-openshift-default-767566b477 from 0 to 1 39m Normal AddedInterface pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Add eth0 [10.133.0.242/23] from ovn-kubernetes 39m Normal AddedInterface pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Add eth0 [10.133.0.241/23] from ovn-kubernetes 39m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Readiness probe error: Get "http://10.133.0.227:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Readiness probe failed: Get "http://10.133.0.227:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Container created 39m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Add eth0 [10.133.0.243/23] from ovn-kubernetes 39m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Container started 39m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc 39m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 39m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj 39m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Container started 39m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Add eth0 [10.134.0.116/23] from ovn-kubernetes 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 39m Normal ScalingReplicaSet deployment/e2e-inf-a-387b16-openshift-default Scaled up replica set e2e-inf-a-387b16-openshift-default-5bb4745897 from 1 to 4 39m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Container created 39m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Add eth0 [10.132.0.124/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Container started 39m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Container created 39m Warning ProbeError pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Readiness probe error: Get "http://10.133.0.229:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Readiness probe failed: Get "http://10.133.0.229:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal ScalingReplicaSet deployment/e2e-inf-b-7ad9ad-openshift-default Scaled up replica set e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 Created pod: e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 39m Normal AddedLabel gateway/e2e-inf-b-7ad9ad Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-7ad9ad 39m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Readiness probe error: Get "http://10.133.0.230:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-c54211 Scaled up replica set payload-processing-e2e-inf-b-c54211-7c65667ccf from 0 to 1 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-de7242-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 39m Normal ScalingReplicaSet deployment/e2e-inf-b-de7242-openshift-default Scaled up replica set e2e-inf-b-de7242-openshift-default-7cf5999677 from 2 to 3 39m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-c54211-7c65667ccf Error creating: pods "payload-processing-e2e-inf-b-c54211-7c65667ccf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-c54211: serviceaccount "payload-processing-e2e-inf-b-c54211" not found 39m Normal SuccessfulCreate replicaset/e2e-inf-b-de7242-openshift-default-7cf5999677 Created pod: e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-c54211 Scaled up replica set payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67 from 0 to 1 39m Normal AddedInterface pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Add eth0 [10.134.0.117/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67 Created pod: payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m 39m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Readiness probe failed: Get "http://10.133.0.230:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-c54211-7c65667ccf Created pod: payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67 Error creating: pods "payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-c54211: serviceaccount "payload-processing-e2e-inf-b-c54211" not found 39m Normal Pulled pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Add eth0 [10.132.0.126/23] from ovn-kubernetes 39m Normal Created pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Container created 39m Normal Started pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Container started 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-62316c Scaled up replica set payload-pre-processing-e2e-inf-b-62316c-544c69559 from 0 to 1 39m Normal Created pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Container created 39m Normal Started pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Container started 39m Normal Created pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Container created 39m Normal Pulled pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-62316c-544c69559 Error creating: pods "payload-pre-processing-e2e-inf-b-62316c-544c69559-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-62316c: serviceaccount "payload-processing-e2e-inf-b-62316c" not found 39m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-62316c-866bb5d4b5 Error creating: pods "payload-processing-e2e-inf-b-62316c-866bb5d4b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-62316c: serviceaccount "payload-processing-e2e-inf-b-62316c" not found 39m Normal Started pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Container started 39m Normal Pulled pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Add eth0 [10.133.0.244/23] from ovn-kubernetes 39m Normal AddedInterface pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Add eth0 [10.133.0.245/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-62316c-544c69559 Created pod: payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-62316c Scaled up replica set payload-processing-e2e-inf-b-62316c-866bb5d4b5 from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-inf-a-c1d09f-openshift-default-67b97686fc Created pod: e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 39m Normal ScalingReplicaSet deployment/e2e-inf-a-c1d09f-openshift-default Scaled up replica set e2e-inf-a-c1d09f-openshift-default-67b97686fc from 1 to 4 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-c1d09f-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 39m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Add eth0 [10.134.0.118/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-62316c-866bb5d4b5 Created pod: payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb 39m Normal SuccessfulCreate replicaset/e2e-inf-a-c1d09f-openshift-default-67b97686fc Created pod: e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl 39m Normal SuccessfulCreate replicaset/e2e-inf-a-c1d09f-openshift-default-67b97686fc Created pod: e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt 39m Normal Pulled pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Container started 39m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Container created 39m Normal AddedInterface pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Add eth0 [10.133.0.246/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Container created 39m Normal Pulled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Startup probe failed: Get "http://10.132.0.124:15021/healthz/ready": dial tcp 10.132.0.124:15021: connect: connection refused 39m Normal Created pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Container created 39m Normal Started pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Container started 39m Normal Created pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Container created 39m Normal AddedInterface pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Add eth0 [10.134.0.120/23] from ovn-kubernetes 39m Normal Started pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Container started 39m Normal Pulled pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Container created 39m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Container started 39m Normal Pulled pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Created pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Container created 39m Normal Started pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Container started 39m Normal Pulled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Container created 39m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Startup probe error: Get "http://10.134.0.116:15021/healthz/ready": dial tcp 10.134.0.116:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Startup probe failed: Get "http://10.134.0.116:15021/healthz/ready": dial tcp 10.134.0.116:15021: connect: connection refused 39m Normal AddedInterface pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Add eth0 [10.134.0.119/23] from ovn-kubernetes 39m Normal Started pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Container started 39m Normal Started pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Container started 39m Normal AddedInterface pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Add eth0 [10.132.0.127/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Pulled pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Startup probe error: Get "http://10.132.0.124:15021/healthz/ready": dial tcp 10.132.0.124:15021: connect: connection refused... 39m Normal Started pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Container started 39m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Startup probe failed: Get "http://10.133.0.243:15021/healthz/ready": dial tcp 10.133.0.243:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Startup probe error: Get "http://10.133.0.243:15021/healthz/ready": dial tcp 10.133.0.243:15021: connect: connection refused... 39m Normal Created pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Container created 39m Normal Pulled pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-7ad9ad-54ff58c567 Error creating: pods "payload-processing-e2e-inf-b-7ad9ad-54ff58c567-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-7ad9ad: serviceaccount "payload-processing-e2e-inf-b-7ad9ad" not found 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-7ad9ad Scaled up replica set payload-processing-e2e-inf-b-7ad9ad-54ff58c567 from 0 to 1 39m Normal Created pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Container created 39m Normal Started pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Container started 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659 Created pod: payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659 Error creating: pods "payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-7ad9ad: serviceaccount "payload-processing-e2e-inf-b-7ad9ad" not found 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-7ad9ad Scaled up replica set payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659 from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-7ad9ad-54ff58c567 Created pod: payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z 39m Normal Created pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Container created 39m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Add eth0 [10.134.0.121/23] from ovn-kubernetes 39m Normal Pulled pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Container started 39m Normal Started pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Container started 39m Normal Created pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Container created 39m Normal Pulled pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Add eth0 [10.134.0.122/23] from ovn-kubernetes 39m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Startup probe failed: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Startup probe error: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused... 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-f78ca5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-f78ca5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Startup probe failed: Get "http://10.134.0.120:15021/healthz/ready": dial tcp 10.134.0.120:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Startup probe error: Get "http://10.134.0.120:15021/healthz/ready": dial tcp 10.134.0.120:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Startup probe failed: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused 39m Warning Unhealthy pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Readiness probe failed: timeout: failed to connect service "10.133.0.55:50052" within 1s: context deadline exceeded 39m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Startup probe error: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused... 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Normal Killing pod/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj Stopping container payload-pre-processing 39m Normal Killing pod/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8 Stopping container payload-processing 39m Warning Unhealthy pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Startup probe failed: Get "http://10.133.0.229:15021/healthz/ready": dial tcp 10.133.0.229:15021: connect: connection refused 39m Warning ProbeError pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Startup probe error: Get "http://10.133.0.229:15021/healthz/ready": dial tcp 10.133.0.229:15021: connect: connection refused... 39m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Startup probe error: Get "http://10.133.0.227:15021/healthz/ready": dial tcp 10.133.0.227:15021: connect: connection refused... 39m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Startup probe error: Get "http://10.133.0.246:15021/healthz/ready": dial tcp 10.133.0.246:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Startup probe failed: Get "http://10.133.0.246:15021/healthz/ready": dial tcp 10.133.0.246:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Startup probe failed: Get "http://10.133.0.227:15021/healthz/ready": dial tcp 10.133.0.227:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Startup probe failed: Get "http://10.133.0.230:15021/healthz/ready": dial tcp 10.133.0.230:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Startup probe error: Get "http://10.133.0.228:15021/healthz/ready": dial tcp 10.133.0.228:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Startup probe failed: Get "http://10.133.0.228:15021/healthz/ready": dial tcp 10.133.0.228:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Startup probe error: Get "http://10.133.0.230:15021/healthz/ready": dial tcp 10.133.0.230:15021: connect: connection refused... 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-c1d09f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-c1d09f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-f78ca5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-f78ca5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-c1d09f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-c1d09f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal ScalingReplicaSet deployment/e2e-inf-a-387b16-openshift-default Scaled up replica set e2e-inf-a-387b16-openshift-default-5bb4745897 from 4 to 7 38m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Add eth0 [10.133.0.249/23] from ovn-kubernetes 38m Normal Pulled pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Add eth0 [10.132.0.130/23] from ovn-kubernetes 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-387b16-openshift-default New size: 7; reason: All metrics below target 38m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q 38m Normal AddedInterface pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Add eth0 [10.133.0.250/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-498mp 38m Normal SuccessfulCreate replicaset/e2e-inf-a-387b16-openshift-default-5bb4745897 Created pod: e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 38m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Container started 38m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Container started 38m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Container created 38m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Container created 38m Normal Created pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Container created 38m Normal Started pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Container started 38m Normal Created pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Container created 38m Normal Started pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Container started 38m Normal Pulled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Startup probe failed: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Startup probe error: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Startup probe failed: Get "http://10.133.0.250:15021/healthz/ready": dial tcp 10.133.0.250:15021: connect: connection refused 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-aee070-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Startup probe error: Get "http://10.133.0.250:15021/healthz/ready": dial tcp 10.133.0.250:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Startup probe failed: Get "http://10.133.0.249:15021/healthz/ready": dial tcp 10.133.0.249:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Startup probe error: Get "http://10.133.0.249:15021/healthz/ready": dial tcp 10.133.0.249:15021: connect: connection refused... 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-aee070-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning ProbeError pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Startup probe error: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Startup probe failed: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused 38m Normal SuccessfulCreate replicaset/e2e-inf-b-0d64c9-openshift-default-585f89bc6f Created pod: e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 38m Normal ScalingReplicaSet deployment/e2e-inf-b-0d64c9-openshift-default Scaled up replica set e2e-inf-b-0d64c9-openshift-default-585f89bc6f from 0 to 1 38m Normal AddedLabel gateway/e2e-inf-b-0d64c9 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-0d64c9 38m Normal Started pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Container started 38m Normal Pulled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Container created 38m Normal Killing pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Stopping container payload-pre-processing 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Startup probe failed: Get "http://10.134.0.113:15021/healthz/ready": dial tcp 10.134.0.113:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Startup probe error: Get "http://10.134.0.113:15021/healthz/ready": dial tcp 10.134.0.113:15021: connect: connection refused... 38m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default Scaled down replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 6 to 5 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 5; reason: All metrics below target 38m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b Stopping container istio-proxy 38m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-xzq5b 38m Normal Killing pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Stopping container istio-proxy 38m Warning Unhealthy pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal AddedInterface pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Add eth0 [10.133.0.254/23] from ovn-kubernetes 38m Warning Unhealthy pod/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Normal Killing pod/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5 Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52 Stopping container payload-pre-processing 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-0d64c9-65ff998c55 Created pod: payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987 Error creating: pods "payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-0d64c9: serviceaccount "payload-processing-e2e-inf-b-0d64c9" not found 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-0d64c9-65ff998c55 Error creating: pods "payload-processing-e2e-inf-b-0d64c9-65ff998c55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-0d64c9: serviceaccount "payload-processing-e2e-inf-b-0d64c9" not found 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-0d64c9 Scaled up replica set payload-processing-e2e-inf-b-0d64c9-65ff998c55 from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-0d64c9 Scaled up replica set payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987 from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987 Created pod: payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 38m Normal Created pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Container created 38m Normal Pulled pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Container created 38m Normal Started pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Container started 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Add eth0 [10.133.1.1/23] from ovn-kubernetes 38m Normal Started pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Container started 38m Normal Pulled pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Add eth0 [10.133.1.0/23] from ovn-kubernetes 37m Warning ProbeError pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Readiness probe error: Get "http://10.133.0.241:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Readiness probe failed: Get "http://10.133.0.242:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning Unhealthy pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Readiness probe failed: Get "http://10.133.0.241:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Readiness probe error: Get "http://10.133.0.242:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Normal Pulled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Container started 37m Normal Pulled pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Container started 37m Normal Created pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Container created 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Created pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Container created 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning Unhealthy pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Readiness probe failed: Get "http://10.134.0.117:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Readiness probe error: Get "http://10.134.0.117:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Pulled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Started pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Container started 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Created pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Container created 37m Warning Unhealthy pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Readiness probe failed: timeout: health rpc did not complete within 1s 37m Warning Unhealthy pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Startup probe failed: Get "http://10.133.0.241:15021/healthz/ready": dial tcp 10.133.0.241:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Startup probe error: Get "http://10.133.0.241:15021/healthz/ready": dial tcp 10.133.0.241:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Startup probe error: Get "http://10.133.0.242:15021/healthz/ready": dial tcp 10.133.0.242:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Startup probe failed: Get "http://10.133.0.242:15021/healthz/ready": dial tcp 10.133.0.242:15021: connect: connection refused 37m Warning Unhealthy pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Startup probe failed: Get "http://10.134.0.117:15021/healthz/ready": dial tcp 10.134.0.117:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Startup probe error: Get "http://10.134.0.117:15021/healthz/ready": dial tcp 10.134.0.117:15021: connect: connection refused... 37m Normal Killing pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr Stopping container istio-proxy 37m Normal Killing pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Stopping container istio-proxy 37m Warning ProbeError pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning Unhealthy pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs Readiness probe failed: HTTP probe failed with statuscode: 503 37m Normal Killing pod/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx Stopping container payload-processing 37m Normal Killing pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Stopping container payload-pre-processing 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 37m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w Stopping container istio-proxy 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 5; reason: All metrics below target 37m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled down replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 6 to 5 37m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w 36m Normal AddedInterface pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Add eth0 [10.133.1.10/23] from ovn-kubernetes 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal Pulled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal SuccessfulCreate replicaset/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff Created pod: e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn 36m Normal ScalingReplicaSet deployment/e2e-inf-b-3cc570-openshift-default Scaled up replica set e2e-inf-b-3cc570-openshift-default-7f6b9c66ff from 1 to 2 36m Normal Started pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Container started 36m Normal Created pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Container created 36m Normal Created pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Container created 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-aee070-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal AddedInterface pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Add eth0 [10.134.0.124/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-inf-a-aee070-openshift-default-5b997d785f Created pod: e2e-inf-a-aee070-openshift-default-5b997d785f-855wt 36m Normal ScalingReplicaSet deployment/e2e-inf-a-aee070-openshift-default Scaled up replica set e2e-inf-a-aee070-openshift-default-5b997d785f from 1 to 2 36m Normal Pulled pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Container started 36m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Startup probe failed: Get "http://10.133.1.10:15021/healthz/ready": dial tcp 10.133.1.10:15021: connect: connection refused 36m Normal Killing pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Stopping container istio-proxy 36m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Startup probe error: Get "http://10.133.1.10:15021/healthz/ready": dial tcp 10.133.1.10:15021: connect: connection refused... 36m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cfe14f-openshift-default Scaled up replica set e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 from 0 to 1 36m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal AddedLabel gateway/e2e-shared-a-w0-cfe14f Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-cfe14f 36m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 Created pod: e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr 36m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-2cpxh Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal AddedInterface pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Add eth0 [10.134.0.125/23] from ovn-kubernetes 36m Warning ProbeError pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Startup probe error: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Startup probe failed: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f Created pod: payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-cfe14f Scaled up replica set payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7 from 0 to 1 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7 Error creating: pods "payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f: serviceaccount "payload-processing-e2e-shared-a-w0-cfe14f" not found 36m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f Error creating: pods "payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f: serviceaccount "payload-processing-e2e-shared-a-w0-cfe14f" not found 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7 Created pod: payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-cfe14f Scaled up replica set payload-processing-e2e-shared-a-w0-cfe14f-544dc775f from 0 to 1 36m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Add eth0 [10.134.0.126/23] from ovn-kubernetes 36m Normal Created pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Container created 36m Normal Started pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Container started 36m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Container started 36m Normal Pulled pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Add eth0 [10.134.0.127/23] from ovn-kubernetes 36m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Container created 36m Normal Killing pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j Stopping container istio-proxy 36m Normal AddedLabel gateway/e2e-inf-a-d3e7e6 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-d3e7e6 36m Normal SuccessfulCreate replicaset/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff Created pod: e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 36m Normal ScalingReplicaSet deployment/e2e-inf-a-d3e7e6-openshift-default Scaled up replica set e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff from 0 to 1 36m Normal AddedInterface pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Add eth0 [10.133.1.12/23] from ovn-kubernetes 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-d3e7e6 Scaled up replica set payload-processing-e2e-inf-a-d3e7e6-5d745d757c from 0 to 1 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf Error creating: pods "payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-d3e7e6: serviceaccount "payload-processing-e2e-inf-a-d3e7e6" not found 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf Created pod: payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-d3e7e6-5d745d757c Error creating: pods "payload-processing-e2e-inf-a-d3e7e6-5d745d757c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-d3e7e6: serviceaccount "payload-processing-e2e-inf-a-d3e7e6" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-d3e7e6 Scaled up replica set payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf from 0 to 1 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-d3e7e6-5d745d757c Created pod: payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 36m Normal Pulled pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Add eth0 [10.134.0.128/23] from ovn-kubernetes 36m Normal Started pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Container started 36m Normal Started pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Container started 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Add eth0 [10.133.1.14/23] from ovn-kubernetes 36m Normal Pulled pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Created pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Container created 36m Normal Created pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Container created 36m Normal Pulled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Container created 36m Normal Started pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Container started 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal ScalingReplicaSet deployment/e2e-inf-b-7ad9ad-openshift-default Scaled up replica set e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 from 1 to 2 36m Normal SuccessfulCreate replicaset/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 Created pod: e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt 36m Normal Pulled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Add eth0 [10.133.1.15/23] from ovn-kubernetes 36m Normal Started pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Container started 36m Normal Created pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Container created 36m Warning ProbeError pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Startup probe error: Get "http://10.133.0.254:15021/healthz/ready": dial tcp 10.133.0.254:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Startup probe failed: Get "http://10.133.0.254:15021/healthz/ready": dial tcp 10.133.0.254:15021: connect: connection refused 36m Warning ProbeError pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Startup probe error: Get "http://10.133.1.15:15021/healthz/ready": dial tcp 10.133.1.15:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Startup probe failed: Get "http://10.133.1.15:15021/healthz/ready": dial tcp 10.133.1.15:15021: connect: connection refused 35m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 4 to 3 35m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-6rrhv 35m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-6rrhv Stopping container discovery 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-96dmr 35m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 Stopping container istio-proxy 35m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-2c9x4 35m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58847f9fc5 from 10 to 8 35m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: All metrics below target 35m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal ScalingReplicaSet deployment/e2e-inf-b-c54211-openshift-default Scaled up replica set e2e-inf-b-c54211-openshift-default-767566b477 from 1 to 2 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal SuccessfulCreate replicaset/e2e-inf-b-c54211-openshift-default-767566b477 Created pod: e2e-inf-b-c54211-openshift-default-767566b477-blgbh 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default Scaled down replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 5 to 4 35m Normal Created pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Container created 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 4; reason: All metrics below target 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 4; reason: All metrics below target 35m Normal Started pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Container started 35m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Stopping container istio-proxy 35m Normal AddedInterface pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Add eth0 [10.132.0.133/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn 35m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct 35m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Stopping container istio-proxy 35m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 4; reason: All metrics below target 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 4; reason: All metrics below target 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 4; reason: All metrics below target 35m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 35m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 Stopping container istio-proxy 35m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x 35m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default Scaled down replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 6 to 4 35m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x 35m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh 35m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default Scaled down replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 5 to 4 35m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k 35m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x Stopping container istio-proxy 35m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7pv8k Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default Scaled down replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 5 to 4 35m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-csjm4 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 4; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default Scaled down replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 5 to 4 35m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-n5w9x Stopping container istio-proxy 35m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-8f9ct Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Readiness probe error: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-lvwxn Readiness probe failed: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Startup probe error: Get "http://10.132.0.133:15021/healthz/ready": dial tcp 10.132.0.133:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Startup probe failed: Get "http://10.132.0.133:15021/healthz/ready": dial tcp 10.132.0.133:15021: connect: connection refused 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-dvbqk 35m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 35m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58847f9fc5 from 8 to 7 35m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default New size: 3; reason: All metrics below target 35m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 35m Warning Unhealthy pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Readiness probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 35m Warning ProbeError pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-t7j72 Readiness probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 35m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Readiness probe error: Get "http://10.134.0.125:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Readiness probe failed: Get "http://10.134.0.125:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Normal ScalingReplicaSet deployment/e2e-inf-b-3cc570-openshift-default Scaled up replica set e2e-inf-b-3cc570-openshift-default-7f6b9c66ff from 2 to 3 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-3cc570-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 35m Normal SuccessfulCreate replicaset/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff Created pod: e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg 35m Normal Created pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Container created 35m Normal Started pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Container started 35m Normal Pulled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Add eth0 [10.132.0.135/23] from ovn-kubernetes 35m Normal Started pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Container started 35m Normal Pulled pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedLabel gateway/e2e-shared-b-w0-03d349 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-03d349 35m Normal Created pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Container created 35m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-03d349-openshift-default Scaled up replica set e2e-shared-b-w0-03d349-openshift-default-cdb65584c from 0 to 1 35m Normal AddedInterface pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Add eth0 [10.134.0.130/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-03d349-openshift-default-cdb65584c Created pod: e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Normal SuccessfulCreate replicaset/e2e-inf-b-0d64c9-openshift-default-585f89bc6f Created pod: e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw 35m Normal ScalingReplicaSet deployment/e2e-inf-b-62316c-openshift-default Scaled up replica set e2e-inf-b-62316c-openshift-default-96996f9c9 from 1 to 2 35m Normal AddedInterface pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Add eth0 [10.134.0.131/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-inf-b-62316c-openshift-default-96996f9c9 Created pod: e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 35m Normal ScalingReplicaSet deployment/e2e-inf-b-0d64c9-openshift-default Scaled up replica set e2e-inf-b-0d64c9-openshift-default-585f89bc6f from 1 to 2 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Normal Created pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Container created 35m Normal Pulled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Add eth0 [10.134.0.132/23] from ovn-kubernetes 35m Normal Started pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Container started 35m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Startup probe error: Get "http://10.132.0.135:15021/healthz/ready": dial tcp 10.132.0.135:15021: connect: connection refused... 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-03d349 Scaled up replica set payload-processing-e2e-shared-b-w0-03d349-77c695ffdf from 0 to 1 35m Normal Started pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Container started 35m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Startup probe failed: Get "http://10.132.0.135:15021/healthz/ready": dial tcp 10.132.0.135:15021: connect: connection refused 35m Normal Created pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Container created 35m Normal Pulled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf Error creating: pods "payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-03d349: serviceaccount "payload-processing-e2e-shared-b-w0-03d349" not found 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67 Error creating: pods "payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-03d349: serviceaccount "payload-processing-e2e-shared-b-w0-03d349" not found 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67 Created pod: payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-03d349 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67 from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf Created pod: payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm 35m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Add eth0 [10.132.0.137/23] from ovn-kubernetes 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Startup probe failed: Get "http://10.134.0.125:15021/healthz/ready": dial tcp 10.134.0.125:15021: connect: connection refused 35m Normal Pulling pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 35m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Startup probe error: Get "http://10.134.0.125:15021/healthz/ready": dial tcp 10.134.0.125:15021: connect: connection refused... 35m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Add eth0 [10.132.0.136/23] from ovn-kubernetes 35m Normal Pulling pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 34m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.337s (2.337s including waiting). Image size: 125317101 bytes. 34m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Container created 34m Normal Started pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Container started 34m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Container started 34m Normal Pulled pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.442s (2.442s including waiting). Image size: 125317101 bytes. 34m Normal Created pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Container created 34m Normal SuccessfulCreate replicaset/e2e-inf-b-0d0152-openshift-default-98d856d9c Created pod: e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 34m Normal ScalingReplicaSet deployment/e2e-inf-b-0d0152-openshift-default Scaled up replica set e2e-inf-b-0d0152-openshift-default-98d856d9c from 0 to 1 34m Normal AddedLabel gateway/e2e-inf-b-0d0152 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-0d0152 34m Normal AddedInterface pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Add eth0 [10.133.1.17/23] from ovn-kubernetes 34m Warning ProbeError pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Startup probe error: Get "http://10.134.0.131:15021/healthz/ready": dial tcp 10.134.0.131:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Startup probe failed: Get "http://10.134.0.131:15021/healthz/ready": dial tcp 10.134.0.131:15021: connect: connection refused 34m Warning Unhealthy pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Startup probe failed: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Startup probe error: Get "http://10.134.0.132:15021/healthz/ready": dial tcp 10.134.0.132:15021: connect: connection refused... 34m Normal ScalingReplicaSet deployment/e2e-inf-a-7df9fa-openshift-default Scaled up replica set e2e-inf-a-7df9fa-openshift-default-b67f7d48f from 1 to 2 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 34m Normal SuccessfulCreate replicaset/e2e-inf-a-7df9fa-openshift-default-b67f7d48f Created pod: e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9 Created pod: payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc 34m Normal Started pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Container started 34m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-0d0152-9fb5f5555 Error creating: pods "payload-processing-e2e-inf-b-0d0152-9fb5f5555-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-0d0152: serviceaccount "payload-processing-e2e-inf-b-0d0152" not found 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-0d0152 Scaled up replica set payload-processing-e2e-inf-b-0d0152-9fb5f5555 from 0 to 1 34m Normal AddedInterface pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Add eth0 [10.133.1.18/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-0d0152-9fb5f5555 Created pod: payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr 34m Normal Pulled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-0d0152 Scaled up replica set payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9 from 0 to 1 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9 Error creating: pods "payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-0d0152: serviceaccount "payload-processing-e2e-inf-b-0d0152" not found 34m Normal Created pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Container created 34m Normal Created pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Container created 34m Normal Created pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Container created 34m Normal AddedInterface pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Add eth0 [10.134.0.134/23] from ovn-kubernetes 34m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Add eth0 [10.134.0.133/23] from ovn-kubernetes 34m Normal Pulled pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Container started 34m Normal Pulled pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Container started 34m Warning ProbeError pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Startup probe error: Get "http://10.133.1.18:15021/healthz/ready": dial tcp 10.133.1.18:15021: connect: connection refused... 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning Unhealthy pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Startup probe failed: Get "http://10.133.1.18:15021/healthz/ready": dial tcp 10.133.1.18:15021: connect: connection refused 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-0d0152-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-0d0152-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal Killing pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Stopping container payload-processing 34m Normal Pulled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Container started 34m Normal Created pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Container created 34m Warning Unhealthy pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Startup probe failed: Get "http://10.133.1.12:15021/healthz/ready": dial tcp 10.133.1.12:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Startup probe error: Get "http://10.133.1.12:15021/healthz/ready": dial tcp 10.133.1.12:15021: connect: connection refused... 33m Normal SuccessfulCreate replicaset/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 Created pod: e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl 33m Normal ScalingReplicaSet deployment/e2e-inf-b-7ad9ad-openshift-default Scaled up replica set e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9 from 2 to 3 33m Normal Pulled pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Add eth0 [10.132.0.140/23] from ovn-kubernetes 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-7ad9ad-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 33m Normal Started pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Container started 33m Normal Created pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Container created 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-7df9fa-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 33m Normal ScalingReplicaSet deployment/e2e-inf-a-7df9fa-openshift-default Scaled up replica set e2e-inf-a-7df9fa-openshift-default-b67f7d48f from 2 to 3 33m Normal SuccessfulCreate replicaset/e2e-inf-a-7df9fa-openshift-default-b67f7d48f Created pod: e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 33m Normal Started pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Container started 33m Normal Created pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Container created 33m Normal Pulled pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Add eth0 [10.132.0.141/23] from ovn-kubernetes 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning Unhealthy pod/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal ScalingReplicaSet deployment/e2e-inf-b-c54211-openshift-default Scaled up replica set e2e-inf-b-c54211-openshift-default-767566b477 from 2 to 3 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal AddedInterface pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Add eth0 [10.134.0.135/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/e2e-inf-b-c54211-openshift-default-767566b477 Created pod: e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-c54211-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 33m Normal Created pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Container created 33m Normal Pulled pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Container started 33m Warning Unhealthy pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Startup probe failed: Get "http://10.132.0.140:15021/healthz/ready": dial tcp 10.132.0.140:15021: connect: connection refused 33m Warning ProbeError pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Startup probe error: Get "http://10.132.0.140:15021/healthz/ready": dial tcp 10.132.0.140:15021: connect: connection refused... 33m Warning ProbeError pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Startup probe error: Get "http://10.132.0.141:15021/healthz/ready": dial tcp 10.132.0.141:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Startup probe failed: Get "http://10.132.0.141:15021/healthz/ready": dial tcp 10.132.0.141:15021: connect: connection refused 33m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused 33m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused... 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal Started pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Container started 33m Normal Created pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Container created 33m Normal Pulled pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Container created 33m Normal Pulled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Container started 33m Warning ProbeError pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Startup probe error: Get "http://10.133.1.17:15021/healthz/ready": dial tcp 10.133.1.17:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Startup probe failed: Get "http://10.133.1.17:15021/healthz/ready": dial tcp 10.133.1.17:15021: connect: connection refused 33m Warning Unhealthy pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Startup probe failed: Get "http://10.134.0.130:15021/healthz/ready": dial tcp 10.134.0.130:15021: connect: connection refused 33m Warning ProbeError pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Startup probe error: Get "http://10.134.0.130:15021/healthz/ready": dial tcp 10.134.0.130:15021: connect: connection refused... 33m Normal Killing pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs Stopping container istio-proxy 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Killing pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Stopping container payload-pre-processing 33m Normal Killing pod/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp Stopping container payload-processing 33m Normal Killing pod/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd Stopping container payload-processing 33m Normal Killing pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Stopping container payload-pre-processing 32m Normal Killing pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Stopping container payload-pre-processing 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-0d0152-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-0d0152-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Stopping container istio-proxy 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal Killing pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Stopping container istio-proxy 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-6jpc2 Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-r4k65 Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning Unhealthy pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Readiness probe failed: Get "http://10.133.0.243:15021/healthz/ready": read tcp 10.133.0.2:45844->10.133.0.243:15021: read: connection reset by peer 32m Warning ProbeError pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj Readiness probe error: Get "http://10.133.0.243:15021/healthz/ready": read tcp 10.133.0.2:45844->10.133.0.243:15021: read: connection reset by peer... 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default New size: 3; reason: All metrics below target 32m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Stopping container istio-proxy 32m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-fml6v Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Stopping container payload-pre-processing 32m Normal Killing pod/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z Stopping container payload-processing 32m Warning Unhealthy pod/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Readiness probe failed: HTTP probe failed with statuscode: 503 32m Normal Killing pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Stopping container istio-proxy 32m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Normal Killing pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Stopping container istio-proxy 32m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn Readiness probe failed: HTTP probe failed with statuscode: 503 32m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cfe14f-openshift-default Scaled up replica set e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 from 1 to 2 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 32m Warning ProbeError pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-dnfgg Readiness probe failed: HTTP probe failed with statuscode: 503 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 Created pod: e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx 32m Normal Pulled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Container created 32m Normal Started pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Container started 32m Normal AddedInterface pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Add eth0 [10.133.1.32/23] from ovn-kubernetes 32m Normal Killing pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Stopping container istio-proxy 32m Warning Unhealthy pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Startup probe failed: Get "http://10.133.1.32:15021/healthz/ready": dial tcp 10.133.1.32:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Startup probe error: Get "http://10.133.1.32:15021/healthz/ready": dial tcp 10.133.1.32:15021: connect: connection refused... 32m Normal ScalingReplicaSet deployment/e2e-inf-b-62316c-openshift-default Scaled up replica set e2e-inf-b-62316c-openshift-default-96996f9c9 from 2 to 3 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-62316c-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 32m Normal SuccessfulCreate replicaset/e2e-inf-b-62316c-openshift-default-96996f9c9 Created pod: e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv 32m Normal AddedInterface pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Add eth0 [10.132.0.144/23] from ovn-kubernetes 32m Normal Created pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Container created 32m Normal Started pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Container started 32m Normal Pulled pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Killing pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Stopping container payload-pre-processing 31m Warning Unhealthy pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Readiness probe failed: HTTP probe failed with statuscode: 503 31m Normal Killing pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Stopping container istio-proxy 31m Warning ProbeError pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Startup probe error: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Startup probe failed: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused 31m Normal Killing pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt Stopping container istio-proxy 31m Warning ProbeError pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Normal Killing pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-r2gjl Stopping container istio-proxy 31m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf Created pod: e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 31m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-d1bfce-openshift-default Scaled up replica set e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf from 0 to 1 31m Normal AddedLabel gateway/e2e-shared-a-w5-d1bfce Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-d1bfce 31m Normal AddedInterface pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Add eth0 [10.133.1.35/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99 Created pod: payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65 Created pod: payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65 Error creating: pods "payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce: serviceaccount "payload-processing-e2e-shared-a-w5-d1bfce" not found 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-d1bfce Scaled up replica set payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65 from 0 to 1 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99 Error creating: pods "payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce: serviceaccount "payload-processing-e2e-shared-a-w5-d1bfce" not found 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-d1bfce Scaled up replica set payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99 from 0 to 1 31m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Container started 31m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Add eth0 [10.133.1.37/23] from ovn-kubernetes 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Add eth0 [10.133.1.38/23] from ovn-kubernetes 31m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Container created 31m Normal Created pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Container created 31m Normal Pulled pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Container started 31m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Warning Unhealthy pod/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 31m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-03d349-openshift-default Scaled up replica set e2e-shared-b-w0-03d349-openshift-default-cdb65584c from 1 to 2 31m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-03d349-openshift-default-cdb65584c Created pod: e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 31m Normal Started pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Container started 31m Normal Created pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Container created 31m Normal AddedInterface pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Add eth0 [10.133.1.40/23] from ovn-kubernetes 31m Normal Pulled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 31m Normal SuccessfulCreate replicaset/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff Created pod: e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf 31m Normal ScalingReplicaSet deployment/e2e-inf-a-d3e7e6-openshift-default Scaled up replica set e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff from 1 to 2 31m Normal Pulled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Container created 31m Normal AddedInterface pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Add eth0 [10.134.0.136/23] from ovn-kubernetes 31m Normal Started pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Container started 31m Warning Unhealthy pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Startup probe failed: Get "http://10.133.1.40:15021/healthz/ready": dial tcp 10.133.1.40:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Startup probe error: Get "http://10.133.1.40:15021/healthz/ready": dial tcp 10.133.1.40:15021: connect: connection refused... 31m Normal Killing pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Stopping container istio-proxy 31m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-24db22-openshift-default Scaled up replica set e2e-shared-a-w1-24db22-openshift-default-879d78fb5 from 0 to 1 31m Warning ProbeError pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Startup probe error: Get "http://10.134.0.136:15021/healthz/ready": dial tcp 10.134.0.136:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Startup probe failed: Get "http://10.134.0.136:15021/healthz/ready": dial tcp 10.134.0.136:15021: connect: connection refused 31m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-24db22-openshift-default-879d78fb5 Created pod: e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l 31m Normal AddedLabel gateway/e2e-shared-a-w1-24db22 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-24db22 31m Normal AddedInterface pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Add eth0 [10.133.1.41/23] from ovn-kubernetes 31m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning ProbeError pod/e2e-inf-b-c54211-openshift-default-767566b477-blgbh Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 Readiness probe failed: HTTP probe failed with statuscode: 503 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-24db22 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795 from 0 to 1 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f Created pod: payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f Error creating: pods "payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-24db22: serviceaccount "payload-processing-e2e-shared-a-w1-24db22" not found 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795 Created pod: payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-24db22: serviceaccount "payload-processing-e2e-shared-a-w1-24db22" not found 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-24db22 Scaled up replica set payload-processing-e2e-shared-a-w1-24db22-68d6bd569f from 0 to 1 31m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Add eth0 [10.133.1.42/23] from ovn-kubernetes 31m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Container created 31m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Container started 31m Normal Pulled pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Add eth0 [10.133.1.43/23] from ovn-kubernetes 31m Normal Created pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Container created 31m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Container started 31m Normal ScalingReplicaSet deployment/e2e-worker-w3-128436-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w3-128436-openshift-default-fd9c97c99 from 3 to 2 31m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw 31m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/e2e-worker-w3-128436-openshift-default-fd9c97c99 Deleted pod: e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-128436-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 31m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Stopping container istio-proxy 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default New size: 3; reason: All metrics below target 31m Warning ProbeError pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Readiness probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 31m Warning Unhealthy pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw Readiness probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 31m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-p99cc 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-xjhtt 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-xfps7 31m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Stopping container istio-proxy 31m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Stopping container istio-proxy 31m Normal Killing pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Stopping container payload-pre-processing 31m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 31m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58847f9fc5 from 7 to 4 31m Normal Killing pod/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp Stopping container payload-processing 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default New size: 3; reason: All metrics below target 31m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz 31m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-a35835-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 31m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 Deleted pod: e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn 31m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Stopping container istio-proxy 31m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc 31m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf Stopping container istio-proxy 31m Normal ScalingReplicaSet deployment/e2e-worker-w5-a35835-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w5-a35835-openshift-default-56c7cfb9d5 from 3 to 2 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default New size: 3; reason: All metrics below target 31m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default New size: 3; reason: All metrics below target 31m Normal Killing pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Stopping container payload-pre-processing 31m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Normal Killing pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Stopping container payload-processing 31m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-p99cc Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-xjhtt Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-xfps7 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-8jvdz Readiness probe failed: HTTP probe failed with statuscode: 503 30m Normal Killing pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mg9rt Stopping container istio-proxy 30m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning Unhealthy pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning ProbeError pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal Created pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Container created 30m Normal Pulled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Warning Unhealthy pod/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal Started pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Container started 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cfe14f-openshift-default Scaled up replica set e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 from 2 to 3 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 Created pod: e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr 30m Normal AddedInterface pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Add eth0 [10.132.0.146/23] from ovn-kubernetes 30m Normal Killing pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Stopping container payload-pre-processing 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-815776-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-815776-openshift-default Scaled up replica set e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9 from 1 to 2 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9 Created pod: e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf 30m Normal Created pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Container created 30m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Startup probe failed: Get "http://10.132.0.146:15021/healthz/ready": dial tcp 10.132.0.146:15021: connect: connection refused 30m Normal Started pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Container started 30m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Startup probe error: Get "http://10.132.0.146:15021/healthz/ready": dial tcp 10.132.0.146:15021: connect: connection refused... 30m Normal AddedInterface pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Add eth0 [10.133.1.50/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal SuccessfulCreate replicaset/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff Created pod: e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-0d64c9-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 30m Normal ScalingReplicaSet deployment/e2e-inf-b-0d64c9-openshift-default Scaled up replica set e2e-inf-b-0d64c9-openshift-default-585f89bc6f from 2 to 3 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-d3e7e6-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 30m Normal ScalingReplicaSet deployment/e2e-inf-a-d3e7e6-openshift-default Scaled up replica set e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff from 2 to 3 30m Normal Started pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Container started 30m Normal Pulled pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Container created 30m Normal AddedInterface pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Add eth0 [10.132.0.148/23] from ovn-kubernetes 30m Normal SuccessfulCreate replicaset/e2e-inf-b-0d64c9-openshift-default-585f89bc6f Created pod: e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q 30m Normal Started pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Container started 30m Normal AddedInterface pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Add eth0 [10.132.0.147/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Container created 30m Normal Killing pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Stopping container istio-proxy 30m Warning ProbeError pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Startup probe error: Get "http://10.133.1.50:15021/healthz/ready": dial tcp 10.133.1.50:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Startup probe failed: Get "http://10.133.1.50:15021/healthz/ready": dial tcp 10.133.1.50:15021: connect: connection refused 30m Normal AddedLabel gateway/e2e-shared-b-w5-91f42c Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-91f42c 30m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-91f42c-openshift-default Scaled up replica set e2e-shared-b-w5-91f42c-openshift-default-b975d7965 from 0 to 1 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-91f42c-openshift-default-b975d7965 Created pod: e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg 30m Warning Unhealthy pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Readiness probe error: HTTP probe failed with statuscode: 503... 30m Normal AddedInterface pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Add eth0 [10.134.0.137/23] from ovn-kubernetes 30m Warning ProbeError pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-t7hr8 Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Startup probe error: Get "http://10.132.0.148:15021/healthz/ready": dial tcp 10.132.0.148:15021: connect: connection refused... 30m Warning ProbeError pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Startup probe error: Get "http://10.132.0.147:15021/healthz/ready": dial tcp 10.132.0.147:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Startup probe failed: Get "http://10.132.0.148:15021/healthz/ready": dial tcp 10.132.0.148:15021: connect: connection refused 30m Warning Unhealthy pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Startup probe failed: Get "http://10.132.0.147:15021/healthz/ready": dial tcp 10.132.0.147:15021: connect: connection refused 30m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-91f42c Scaled up replica set payload-processing-e2e-shared-b-w5-91f42c-c8bd94999 from 0 to 1 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999 Error creating: pods "payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c: serviceaccount "payload-processing-e2e-shared-b-w5-91f42c" not found 30m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f Created pod: payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 30m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-91f42c Scaled up replica set payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f from 0 to 1 30m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999 Created pod: payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg 30m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f Error creating: pods "payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c: serviceaccount "payload-processing-e2e-shared-b-w5-91f42c" not found 30m Warning Unhealthy pod/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal Pulled pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal Created pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Container created 30m Normal Started pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Container started 30m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Readiness probe error: Get "http://10.133.1.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 30m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Container created 30m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Container started 30m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Add eth0 [10.134.0.138/23] from ovn-kubernetes 30m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Add eth0 [10.134.0.139/23] from ovn-kubernetes 30m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Readiness probe failed: Get "http://10.133.1.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 30m Normal Killing pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Stopping container payload-processing 30m Normal Started pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Container started 30m Normal Pulled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Container created 30m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Startup probe error: Get "http://10.133.1.35:15021/healthz/ready": dial tcp 10.133.1.35:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Startup probe failed: Get "http://10.133.1.35:15021/healthz/ready": dial tcp 10.133.1.35:15021: connect: connection refused 30m Normal AddedInterface pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Add eth0 [10.134.0.140/23] from ovn-kubernetes 30m Normal Started pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Container started 30m Normal SuccessfulCreate replicaset/e2e-inf-b-0d0152-openshift-default-98d856d9c Created pod: e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-0d0152-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal Pulled pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal ScalingReplicaSet deployment/e2e-inf-b-0d0152-openshift-default Scaled up replica set e2e-inf-b-0d0152-openshift-default-98d856d9c from 1 to 2 30m Normal Created pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Container created 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal Killing pod/e2e-inf-b-62316c-openshift-default-96996f9c9-wntgv Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 Stopping container istio-proxy 30m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-03d349-openshift-default Scaled up replica set e2e-shared-b-w0-03d349-openshift-default-cdb65584c from 2 to 3 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 30m Normal Pulled pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Add eth0 [10.132.0.149/23] from ovn-kubernetes 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-03d349-openshift-default-cdb65584c Created pod: e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw 30m Normal Started pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Container started 30m Normal Created pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Container created 30m Warning ProbeError pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Startup probe error: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused... 30m Normal Killing pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9q94q Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw Stopping container istio-proxy 30m Warning Unhealthy pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Startup probe failed: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Startup probe error: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Startup probe failed: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused 29m Normal AddedLabel gateway/e2e-shared-b-w1-0f980e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-0f980e 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-0f980e-openshift-default Scaled up replica set e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 from 0 to 1 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc 29m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning Unhealthy pod/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Add eth0 [10.133.1.56/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-0f980e Scaled up replica set payload-processing-e2e-shared-b-w1-0f980e-b878b8d69 from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-0f980e Scaled up replica set payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7 from 0 to 1 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7 Error creating: pods "payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e: serviceaccount "payload-processing-e2e-shared-b-w1-0f980e" not found 29m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69 Error creating: pods "payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e: serviceaccount "payload-processing-e2e-shared-b-w1-0f980e" not found 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69 Created pod: payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7 Created pod: payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz 29m Normal Created pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Container created 29m Normal Pulled pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Container started 29m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Container created 29m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Add eth0 [10.133.1.57/23] from ovn-kubernetes 29m Normal Started pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Container started 29m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Add eth0 [10.133.1.58/23] from ovn-kubernetes 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Normal Pulled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Container created 29m Normal Started pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Container started 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-cc33d3-openshift-default Scaled up replica set e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 from 1 to 2 29m Normal Created pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Container created 29m Normal Pulled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Warning ProbeError pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Startup probe error: Get "http://10.133.1.41:15021/healthz/ready": dial tcp 10.133.1.41:15021: connect: connection refused... 29m Normal Started pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Container started 29m Normal AddedInterface pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Add eth0 [10.133.1.63/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Created pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 29m Warning Unhealthy pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Startup probe failed: Get "http://10.133.1.41:15021/healthz/ready": dial tcp 10.133.1.41:15021: connect: connection refused 29m Normal AddedLabel gateway/e2e-ipp-a-db0540 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-db0540 29m Normal AddedInterface pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Add eth0 [10.134.0.141/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/e2e-ipp-a-db0540-openshift-default-5d778c79bf Created pod: e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 29m Normal ScalingReplicaSet deployment/e2e-ipp-a-db0540-openshift-default Scaled up replica set e2e-ipp-a-db0540-openshift-default-5d778c79bf from 0 to 1 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 Created pod: e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 29m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Startup probe error: Get "http://10.133.1.63:15021/healthz/ready": dial tcp 10.133.1.63:15021: connect: connection refused... 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cfe14f-openshift-default Scaled up replica set e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 from 3 to 4 29m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Startup probe failed: Get "http://10.133.1.63:15021/healthz/ready": dial tcp 10.133.1.63:15021: connect: connection refused 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-6b5b1e-openshift-default Scaled up replica set e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 from 1 to 2 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-db0540 Scaled up replica set payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f Created pod: payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f Error creating: pods "payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-db0540: serviceaccount "payload-processing-e2e-ipp-a-db0540" not found 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7 Error creating: pods "payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-db0540: serviceaccount "payload-processing-e2e-ipp-a-db0540" not found 29m Normal AddedInterface pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Add eth0 [10.134.0.142/23] from ovn-kubernetes 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 Created pod: e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj 29m Normal Pulled pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Container created 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-db0540 Scaled up replica set payload-processing-e2e-ipp-a-db0540-c8f7b8fd7 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7 Created pod: payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp 29m Normal Started pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Container started 29m Normal AddedInterface pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Add eth0 [10.133.1.65/23] from ovn-kubernetes 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Container created 29m Normal Started pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Container started 29m Normal Pulled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Container created 29m Normal Started pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Container started 29m Normal Pulled pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Add eth0 [10.133.1.66/23] from ovn-kubernetes 29m Normal Started pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Container started 29m Normal Created pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Container created 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Add eth0 [10.134.0.143/23] from ovn-kubernetes 29m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Startup probe failed: Get "http://10.133.1.66:15021/healthz/ready": dial tcp 10.133.1.66:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Startup probe error: Get "http://10.133.1.66:15021/healthz/ready": dial tcp 10.133.1.66:15021: connect: connection refused... 29m Normal Killing pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Stopping container payload-pre-processing 28m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled down replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 2 to 1 28m Normal SuccessfulDelete replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Deleted pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default New size: 1; reason: All metrics below target 28m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning Unhealthy pod/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal Killing pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 Stopping container istio-proxy 28m Warning Unhealthy pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning Unhealthy pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-c9zh8 Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning ProbeError pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal Started pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Container started 28m Normal Created pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Container created 28m Normal Pulled pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Container created 28m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Container started 28m Normal Killing pod/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Stopping container payload-pre-processing 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning ProbeError pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Startup probe error: Get "http://10.134.0.137:15021/healthz/ready": dial tcp 10.134.0.137:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Startup probe failed: Get "http://10.134.0.137:15021/healthz/ready": dial tcp 10.134.0.137:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Startup probe failed: Get "http://10.133.1.56:15021/healthz/ready": dial tcp 10.133.1.56:15021: connect: connection refused 28m Normal AddedInterface pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Add eth0 [10.134.0.144/23] from ovn-kubernetes 28m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-477f6b-openshift-default-7854c78755 Created pod: e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 28m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Startup probe error: Get "http://10.133.1.56:15021/healthz/ready": dial tcp 10.133.1.56:15021: connect: connection refused... 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-477f6b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 28m Normal Pulled pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-477f6b-openshift-default Scaled up replica set e2e-shared-a-w6-477f6b-openshift-default-7854c78755 from 1 to 2 28m Normal Created pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Container created 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-91f42c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-91f42c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal Started pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Container started 28m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled up replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 1 to 2 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 28m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 28m Warning ProbeError pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Startup probe error: Get "http://10.134.0.144:15021/healthz/ready": dial tcp 10.134.0.144:15021: connect: connection refused... 28m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Add eth0 [10.133.1.73/23] from ovn-kubernetes 28m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Container started 28m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Container created 28m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Startup probe failed: Get "http://10.134.0.144:15021/healthz/ready": dial tcp 10.134.0.144:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Startup probe error: Get "http://10.133.1.73:15021/healthz/ready": dial tcp 10.133.1.73:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Startup probe failed: Get "http://10.133.1.73:15021/healthz/ready": dial tcp 10.133.1.73:15021: connect: connection refused 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-91f42c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-91f42c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk Stopping container payload-pre-processing 27m Normal Killing pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 Stopping container istio-proxy 27m Normal Killing pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq Stopping container istio-proxy 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7 Created pod: e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-25d881-openshift-default Scaled up replica set e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7 from 0 to 1 27m Normal AddedLabel gateway/e2e-shared-a-w2-25d881 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-25d881 27m Normal AddedInterface pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Add eth0 [10.134.0.145/23] from ovn-kubernetes 27m Normal AddedLabel gateway/e2e-ipp-a-dddc53 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-dddc53 27m Normal ScalingReplicaSet deployment/e2e-ipp-a-dddc53-openshift-default Scaled up replica set e2e-ipp-a-dddc53-openshift-default-69ddd67fc from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-ipp-a-dddc53-openshift-default-69ddd67fc Created pod: e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 27m Normal AddedInterface pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Add eth0 [10.133.1.74/23] from ovn-kubernetes 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-25d881 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599 from 0 to 1 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599 Error creating: pods "payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-25d881: serviceaccount "payload-processing-e2e-shared-a-w2-25d881" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599 Created pod: payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c Created pod: payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-25d881 Scaled up replica set payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c from 0 to 1 27m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c Error creating: pods "payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-25d881: serviceaccount "payload-processing-e2e-shared-a-w2-25d881" not found 27m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Container created 27m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Container created 27m Normal Pulled pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Add eth0 [10.134.0.146/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Add eth0 [10.133.1.75/23] from ovn-kubernetes 27m Normal Started pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Container started 27m Warning Unhealthy pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Readiness probe failed: Get "http://10.134.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning ProbeError pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Readiness probe error: Get "http://10.134.0.141:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-dddc53-55b47855ff Error creating: pods "payload-processing-e2e-ipp-a-dddc53-55b47855ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dddc53: serviceaccount "payload-processing-e2e-ipp-a-dddc53" not found 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Add eth0 [10.134.0.147/23] from ovn-kubernetes 27m Normal Created pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Container created 27m Normal Pulled pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-dddc53 Scaled up replica set payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c from 0 to 1 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-dddc53 Scaled up replica set payload-processing-e2e-ipp-a-dddc53-55b47855ff from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c Created pod: payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-dddc53-55b47855ff Created pod: payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr 27m Normal Started pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Container started 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c Error creating: pods "payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dddc53: serviceaccount "payload-processing-e2e-ipp-a-dddc53" not found 27m Normal Started pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Container created 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Add eth0 [10.134.0.148/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Container started 27m Normal Created pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Container created 27m Normal Pulled pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-d1bfce-openshift-default Scaled up replica set e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf from 1 to 2 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf Created pod: e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 27m Normal Created pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Container created 27m Normal AddedInterface pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Add eth0 [10.134.0.149/23] from ovn-kubernetes 27m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 27m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 27m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-xfmtk 27m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk Stopping container discovery 27m Normal Pulled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Container started 27m Warning Unhealthy pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Startup probe failed: Get "http://10.134.0.141:15021/healthz/ready": dial tcp 10.134.0.141:15021: connect: connection refused 27m Warning ProbeError pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Startup probe error: Get "http://10.134.0.141:15021/healthz/ready": dial tcp 10.134.0.141:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Startup probe error: Get "http://10.134.0.149:15021/healthz/ready": dial tcp 10.134.0.149:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Startup probe failed: Get "http://10.134.0.149:15021/healthz/ready": dial tcp 10.134.0.149:15021: connect: connection refused 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-f22a52-openshift-default Scaled up replica set e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f from 1 to 2 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f Created pod: e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-db0540 Scaled up replica set payload-processing-e2e-ipp-a-db0540-85875d5ff6 from 0 to 1 27m Normal Started pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Container started 27m Normal Created pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Container created 27m Normal AddedInterface pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Add eth0 [10.134.0.150/23] from ovn-kubernetes 27m Normal Pulled pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-24db22-openshift-default Scaled up replica set e2e-shared-a-w1-24db22-openshift-default-879d78fb5 from 1 to 2 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-db0540 Scaled up replica set payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7 from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-24db22-openshift-default-879d78fb5 Created pod: e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-db0540-85875d5ff6 Created pod: payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-db0540-85875d5ff6 Error creating: pods "payload-processing-e2e-ipp-a-db0540-85875d5ff6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-db0540: serviceaccount "payload-processing-e2e-ipp-a-db0540" not found 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7 Error creating: pods "payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-db0540: serviceaccount "payload-processing-e2e-ipp-a-db0540" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7 Created pod: payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj 27m Normal AddedInterface pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Add eth0 [10.134.0.151/23] from ovn-kubernetes 27m Normal Pulling pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 27m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 27m Normal Created pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Container created 27m Normal Started pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Container started 27m Normal Pulled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Add eth0 [10.132.0.157/23] from ovn-kubernetes 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Add eth0 [10.132.0.156/23] from ovn-kubernetes 27m Normal Started pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Container started 27m Normal Pulled pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.116s (1.116s including waiting). Image size: 166612641 bytes. 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Container started 27m Normal Created pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Container created 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Container created 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.281s (1.281s including waiting). Image size: 166612641 bytes. 27m Warning ProbeError pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Startup probe error: Get "http://10.134.0.150:15021/healthz/ready": dial tcp 10.134.0.150:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Startup probe failed: Get "http://10.134.0.150:15021/healthz/ready": dial tcp 10.134.0.150:15021: connect: connection refused 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-db0540-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-db0540-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning ProbeError pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Startup probe error: Get "http://10.134.0.151:15021/healthz/ready": dial tcp 10.134.0.151:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Startup probe failed: Get "http://10.134.0.151:15021/healthz/ready": dial tcp 10.134.0.151:15021: connect: connection refused 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-2d4576-openshift-default-779b666c6f Created pod: e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw 27m Normal AddedLabel gateway/e2e-ipp-b-2d4576 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-2d4576 27m Normal ScalingReplicaSet deployment/e2e-ipp-b-2d4576-openshift-default Scaled up replica set e2e-ipp-b-2d4576-openshift-default-779b666c6f from 0 to 1 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-db0540-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal AddedInterface pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Add eth0 [10.134.0.153/23] from ovn-kubernetes 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-db0540-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7 Created pod: payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-2d4576 Scaled up replica set payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7 from 0 to 1 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-2d4576 Scaled up replica set payload-processing-e2e-ipp-b-2d4576-5467db566c from 0 to 1 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7 Error creating: pods "payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2d4576: serviceaccount "payload-processing-e2e-ipp-b-2d4576" not found 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-2d4576-5467db566c Created pod: payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-2d4576-5467db566c Error creating: pods "payload-processing-e2e-ipp-b-2d4576-5467db566c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2d4576: serviceaccount "payload-processing-e2e-ipp-b-2d4576" not found 26m Normal AddedInterface pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Add eth0 [10.133.1.83/23] from ovn-kubernetes 26m Normal Started pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Container started 26m Normal Created pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Container created 26m Normal Started pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Container started 26m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Container created 26m Normal Pulled pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Add eth0 [10.134.0.154/23] from ovn-kubernetes 26m Normal SuccessfulDelete replicaset/e2e-worker-w2-c78066-openshift-default-5696c477f7 Deleted pod: e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl 26m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Stopping container istio-proxy 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c78066-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-e2e3d0-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 26m Normal SuccessfulDelete replicaset/e2e-worker-w4-e2e3d0-openshift-default-57898b578c Deleted pod: e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn 26m Normal ScalingReplicaSet deployment/e2e-worker-w4-e2e3d0-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w4-e2e3d0-openshift-default-57898b578c from 3 to 2 26m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Stopping container istio-proxy 26m Normal ScalingReplicaSet deployment/e2e-worker-w2-c78066-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-c78066-openshift-default-5696c477f7 from 3 to 2 26m Normal AddedLabel gateway/e2e-ipp-a-bf87e1 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-bf87e1 26m Normal ScalingReplicaSet deployment/e2e-ipp-a-bf87e1-openshift-default Scaled up replica set e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf from 0 to 1 26m Normal AddedInterface pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Add eth0 [10.133.1.84/23] from ovn-kubernetes 26m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Readiness probe error: HTTP probe failed with statuscode: 503... 26m Warning ProbeError pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Readiness probe error: HTTP probe failed with statuscode: 503... 26m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf Created pod: e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm 26m Warning Unhealthy pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl Readiness probe failed: HTTP probe failed with statuscode: 503 26m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn Readiness probe failed: HTTP probe failed with statuscode: 503 26m Normal AddedLabel gateway/e2e-ipp-a-dfc3d2 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-dfc3d2 26m Normal SuccessfulCreate replicaset/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f Created pod: e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-bf87e1 Scaled up replica set payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5 from 0 to 1 26m Normal ScalingReplicaSet deployment/e2e-ipp-a-dfc3d2-openshift-default Scaled up replica set e2e-ipp-a-dfc3d2-openshift-default-765b549c9f from 0 to 1 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-bf87e1 Scaled up replica set payload-processing-e2e-ipp-a-bf87e1-cc959cd95 from 0 to 1 26m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-bf87e1-cc959cd95 Error creating: pods "payload-processing-e2e-ipp-a-bf87e1-cc959cd95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf87e1: serviceaccount "payload-processing-e2e-ipp-a-bf87e1" not found 26m Normal AddedInterface pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Add eth0 [10.133.1.85/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5 Created pod: payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5 Error creating: pods "payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf87e1: serviceaccount "payload-processing-e2e-ipp-a-bf87e1" not found 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-bf87e1-cc959cd95 Created pod: payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 26m Normal Started pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Container started 26m Normal Created pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Container created 26m Normal Pulled pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Add eth0 [10.133.1.86/23] from ovn-kubernetes 26m Normal Started pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Container started 26m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Add eth0 [10.134.0.155/23] from ovn-kubernetes 26m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Container created 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99 Error creating: pods "payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2: serviceaccount "payload-processing-e2e-ipp-a-dfc3d2" not found 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99 Created pod: payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-dfc3d2 Scaled up replica set payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99 from 0 to 1 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7 Created pod: payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-dfc3d2 Scaled up replica set payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7 from 0 to 1 26m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7 Error creating: pods "payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2: serviceaccount "payload-processing-e2e-ipp-a-dfc3d2" not found 26m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Add eth0 [10.133.1.88/23] from ovn-kubernetes 26m Normal Started pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Container started 26m Normal AddedInterface pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Add eth0 [10.133.1.87/23] from ovn-kubernetes 26m Normal Started pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Container started 26m Normal Created pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Container created 26m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Container created 26m Normal Pulled pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-f6302c-openshift-default Scaled up replica set e2e-shared-b-w2-f6302c-openshift-default-775744b6b9 from 0 to 1 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9 Created pod: e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn 26m Normal AddedLabel gateway/e2e-shared-b-w2-f6302c Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-f6302c 26m Normal AddedInterface pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Add eth0 [10.134.0.156/23] from ovn-kubernetes 26m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455 Error creating: pods "payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c: serviceaccount "payload-processing-e2e-shared-b-w2-f6302c" not found 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-f6302c Scaled up replica set payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc from 0 to 1 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc Created pod: payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455 Created pod: payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-f6302c Scaled up replica set payload-processing-e2e-shared-b-w2-f6302c-5b5674d455 from 0 to 1 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc Error creating: pods "payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c: serviceaccount "payload-processing-e2e-shared-b-w2-f6302c" not found 26m Normal Created pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Container created 26m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Add eth0 [10.133.1.100/23] from ovn-kubernetes 26m Normal Started pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Container started 26m Normal Pulled pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Warning Unhealthy pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Readiness probe failed: Get "http://10.134.0.145:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Add eth0 [10.134.0.165/23] from ovn-kubernetes 26m Warning ProbeError pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Readiness probe error: Get "http://10.134.0.145:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Container created 26m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Container started 26m Normal Killing pod/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr Stopping container payload-processing 26m Normal Pulled pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Container started 26m Normal Created pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Container created 26m Warning ProbeError pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Readiness probe error: Get "http://10.133.1.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Readiness probe failed: Get "http://10.133.1.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Startup probe failed: Get "http://10.134.0.145:15021/healthz/ready": dial tcp 10.134.0.145:15021: connect: connection refused 26m Normal Pulled pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Container created 26m Normal Started pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Container started 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j 26m Warning ProbeError pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Startup probe error: Get "http://10.134.0.145:15021/healthz/ready": dial tcp 10.134.0.145:15021: connect: connection refused... 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-0f980e-openshift-default Scaled up replica set e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 from 1 to 2 26m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Container started 26m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Container created 26m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Add eth0 [10.134.0.166/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 Created pod: e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-6b5b1e-openshift-default Scaled up replica set e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 from 2 to 3 26m Normal Created pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Container created 26m Normal Pulled pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Container started 26m Normal AddedInterface pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Add eth0 [10.132.0.165/23] from ovn-kubernetes 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-25d881-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf Created pod: e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-d1bfce-openshift-default Scaled up replica set e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf from 2 to 3 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-25d881-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Normal Started pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Container started 26m Normal AddedInterface pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Add eth0 [10.132.0.166/23] from ovn-kubernetes 26m Normal Created pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Container created 26m Normal Pulled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning Unhealthy pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Startup probe failed: Get "http://10.133.1.74:15021/healthz/ready": dial tcp 10.133.1.74:15021: connect: connection refused 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 26m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Startup probe failed: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Created pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv 26m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Startup probe error: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused... 26m Warning ProbeError pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Startup probe error: Get "http://10.133.1.74:15021/healthz/ready": dial tcp 10.133.1.74:15021: connect: connection refused... 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled up replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 2 to 3 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Created pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx 26m Normal Pulled pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Created pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs 26m Normal Created pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Container created 26m Normal AddedInterface pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Add eth0 [10.134.0.167/23] from ovn-kubernetes 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-cc33d3-openshift-default Scaled up replica set e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 from 2 to 4 26m Normal AddedInterface pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Add eth0 [10.132.0.167/23] from ovn-kubernetes 26m Warning Unhealthy pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Startup probe failed: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused 26m Warning ProbeError pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Startup probe error: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused... 26m Normal AddedInterface pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Add eth0 [10.133.1.101/23] from ovn-kubernetes 26m Normal Started pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Container started 26m Normal Pulled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Container created 26m Normal Pulled pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Container started 26m Normal Started pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Container started 26m Normal Created pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Container created 26m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Startup probe error: Get "http://10.132.0.166:15021/healthz/ready": dial tcp 10.132.0.166:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Startup probe failed: Get "http://10.132.0.166:15021/healthz/ready": dial tcp 10.132.0.166:15021: connect: connection refused 25m Warning Unhealthy pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Startup probe failed: Get "http://10.134.0.167:15021/healthz/ready": dial tcp 10.134.0.167:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Startup probe error: Get "http://10.134.0.167:15021/healthz/ready": dial tcp 10.134.0.167:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Startup probe failed: Get "http://10.132.0.167:15021/healthz/ready": dial tcp 10.132.0.167:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Startup probe error: Get "http://10.132.0.167:15021/healthz/ready": dial tcp 10.132.0.167:15021: connect: connection refused... 25m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Startup probe error: Get "http://10.133.1.101:15021/healthz/ready": dial tcp 10.133.1.101:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Startup probe failed: Get "http://10.133.1.101:15021/healthz/ready": dial tcp 10.133.1.101:15021: connect: connection refused 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-25d881-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-25d881-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-dddc53-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-dddc53-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95 Created pod: payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95 Error creating: pods "payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dddc53: serviceaccount "payload-processing-e2e-ipp-a-dddc53" not found 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-dddc53 Scaled up replica set payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95 from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-dddc53 Scaled up replica set payload-processing-e2e-ipp-a-dddc53-9b584966d from 0 to 1 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-dddc53-9b584966d Created pod: payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-dddc53-9b584966d Error creating: pods "payload-processing-e2e-ipp-a-dddc53-9b584966d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dddc53: serviceaccount "payload-processing-e2e-ipp-a-dddc53" not found 25m Normal AddedInterface pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Add eth0 [10.133.1.104/23] from ovn-kubernetes 25m Normal Pulling pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 25m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Add eth0 [10.133.1.103/23] from ovn-kubernetes 25m Normal Pulled pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.371s (2.371s including waiting). Image size: 166612641 bytes. 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-dddc53-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-dddc53-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.468s (2.468s including waiting). Image size: 166612641 bytes. 25m Normal Created pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Container created 25m Normal Started pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Container started 25m Normal Started pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Container started 25m Normal Created pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Container created 25m Warning Unhealthy pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Readiness probe failed: Get "http://10.134.0.153:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Warning ProbeError pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Readiness probe error: Get "http://10.134.0.153:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Normal Pulled pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Container started 25m Normal Created pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Container created 25m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-24db22-openshift-default-879d78fb5 Created pod: e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 25m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-24db22-openshift-default Scaled up replica set e2e-shared-a-w1-24db22-openshift-default-879d78fb5 from 2 to 3 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-24db22-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 25m Normal Started pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Container started 25m Normal Created pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Container created 25m Normal Pulled pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning Unhealthy pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Startup probe failed: Get "http://10.134.0.153:15021/healthz/ready": dial tcp 10.134.0.153:15021: connect: connection refused 25m Normal AddedInterface pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Add eth0 [10.132.0.169/23] from ovn-kubernetes 25m Warning ProbeError pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Startup probe error: Get "http://10.134.0.153:15021/healthz/ready": dial tcp 10.134.0.153:15021: connect: connection refused... 25m Normal Killing pod/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl Stopping container payload-pre-processing 25m Normal AddedLabel gateway/e2e-ipp-b-f7ad21 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-f7ad21 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-2d4576-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-2d4576-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Normal SuccessfulCreate replicaset/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f Created pod: e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t 25m Normal ScalingReplicaSet deployment/e2e-ipp-b-f7ad21-openshift-default Scaled up replica set e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f from 0 to 1 25m Normal AddedInterface pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Add eth0 [10.134.0.169/23] from ovn-kubernetes 25m Warning Unhealthy pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Startup probe failed: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Startup probe error: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused... 25m Normal ScalingReplicaSet deployment/e2e-worker-w0-197e9b-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w0-197e9b-openshift-default-5784796cff from 3 to 2 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-197e9b-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/e2e-worker-w0-197e9b-openshift-default-5784796cff Deleted pod: e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 25m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Stopping container istio-proxy 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65 Created pod: payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-f7ad21 Scaled up replica set payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc from 0 to 1 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc Error creating: pods "payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f7ad21: serviceaccount "payload-processing-e2e-ipp-b-f7ad21" not found 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65 Error creating: pods "payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f7ad21: serviceaccount "payload-processing-e2e-ipp-b-f7ad21" not found 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc Created pod: payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-f7ad21 Scaled up replica set payload-processing-e2e-ipp-b-f7ad21-6b6957bd65 from 0 to 1 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Add eth0 [10.133.1.107/23] from ovn-kubernetes 25m Normal Started pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Container started 25m Normal AddedInterface pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Add eth0 [10.133.1.108/23] from ovn-kubernetes 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-91f42c-openshift-default-b975d7965 Created pod: e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq 25m Normal Created pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Container created 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-91f42c-openshift-default Scaled up replica set e2e-shared-b-w5-91f42c-openshift-default-b975d7965 from 1 to 2 25m Normal Pulled pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-91f42c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 25m Normal Created pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Container created 25m Normal Started pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Container started 25m Normal Started pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Container started 25m Normal AddedInterface pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Add eth0 [10.133.1.109/23] from ovn-kubernetes 25m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 Readiness probe failed: HTTP probe failed with statuscode: 503 25m Normal Created pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Container created 25m Normal Pulled pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning Unhealthy pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Startup probe failed: Get "http://10.133.1.109:15021/healthz/ready": dial tcp 10.133.1.109:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Startup probe error: Get "http://10.133.1.109:15021/healthz/ready": dial tcp 10.133.1.109:15021: connect: connection refused... 25m Normal Killing pod/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2 Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7 Stopping container payload-processing 24m Normal Pulled pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Container started 24m Normal Created pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Container created 24m Normal Started pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Container started 24m Normal Pulled pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Container created 24m Normal Pulled pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Container started 24m Normal Created pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Container created 24m Normal Killing pod/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2 Stopping container payload-processing 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-2d4576-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-2d4576-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Normal Killing pod/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl Stopping container payload-pre-processing 24m Warning Unhealthy pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Startup probe failed: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Startup probe error: Get "http://10.133.1.85:15021/healthz/ready": dial tcp 10.133.1.85:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Startup probe failed: Get "http://10.133.1.85:15021/healthz/ready": dial tcp 10.133.1.85:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Startup probe failed: Get "http://10.133.1.84:15021/healthz/ready": dial tcp 10.133.1.84:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Startup probe error: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Startup probe error: Get "http://10.133.1.84:15021/healthz/ready": dial tcp 10.133.1.84:15021: connect: connection refused... 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-f6302c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-bf87e1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-bf87e1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-f6302c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-dfc3d2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-dfc3d2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-2d4576 Scaled up replica set payload-processing-e2e-ipp-b-2d4576-755f4cf4dd from 0 to 1 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c Error creating: pods "payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2d4576: serviceaccount "payload-processing-e2e-ipp-b-2d4576" not found 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd Error creating: pods "payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2d4576: serviceaccount "payload-processing-e2e-ipp-b-2d4576" not found 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-2d4576 Scaled up replica set payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c Created pod: payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd Created pod: payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 24m Normal Created pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Container created 24m Normal Started pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Container started 24m Normal Started pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Container started 24m Normal Pulled pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Add eth0 [10.133.1.112/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Container created 24m Normal AddedInterface pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Add eth0 [10.133.1.113/23] from ovn-kubernetes 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-f6302c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-f6302c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-bf87e1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-bf87e1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-dfc3d2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-dfc3d2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7 Error creating: pods "payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf87e1: serviceaccount "payload-processing-e2e-ipp-a-bf87e1" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-bf87e1 Scaled up replica set payload-processing-e2e-ipp-a-bf87e1-5d55c5689b from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-bf87e1 Scaled up replica set payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7 from 0 to 1 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b Error creating: pods "payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf87e1: serviceaccount "payload-processing-e2e-ipp-a-bf87e1" not found 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7 Created pod: payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b Created pod: payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn 24m Normal AddedInterface pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Add eth0 [10.133.1.117/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Add eth0 [10.133.1.116/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Container created 24m Normal Started pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Container started 24m Normal Started pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Container started 24m Normal Created pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Container created 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 23m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb Created pod: payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb Error creating: pods "payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2: serviceaccount "payload-processing-e2e-ipp-a-dfc3d2" not found 23m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-0f980e-openshift-default Scaled up replica set e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 from 2 to 3 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-dfc3d2 Scaled up replica set payload-processing-e2e-ipp-a-dfc3d2-74d558bbb from 0 to 1 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-dfc3d2 Scaled up replica set payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c Created pod: payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c Error creating: pods "payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2: serviceaccount "payload-processing-e2e-ipp-a-dfc3d2" not found 23m Normal Started pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Container started 23m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Add eth0 [10.132.0.170/23] from ovn-kubernetes 23m Normal AddedInterface pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Add eth0 [10.133.1.119/23] from ovn-kubernetes 23m Normal Created pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Container created 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Add eth0 [10.133.1.118/23] from ovn-kubernetes 23m Normal Created pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Container created 23m Normal Started pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Container started 23m Normal Pulled pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Container started 23m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Container created 23m Normal AddedLabel gateway/e2e-ipp-b-d51cd8 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-d51cd8 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf 23m Normal ScalingReplicaSet deployment/e2e-ipp-b-d51cd8-openshift-default Scaled up replica set e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c from 0 to 1 23m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Add eth0 [10.134.0.172/23] from ovn-kubernetes 23m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Startup probe error: Get "http://10.132.0.170:15021/healthz/ready": dial tcp 10.132.0.170:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Startup probe failed: Get "http://10.132.0.170:15021/healthz/ready": dial tcp 10.132.0.170:15021: connect: connection refused 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-d51cd8 Scaled up replica set payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f from 0 to 1 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-d51cd8 Scaled up replica set payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999 from 0 to 1 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f Error creating: pods "payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-d51cd8: serviceaccount "payload-processing-e2e-ipp-b-d51cd8" not found 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f Created pod: payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999 Created pod: payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999 Error creating: pods "payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-d51cd8: serviceaccount "payload-processing-e2e-ipp-b-d51cd8" not found 23m Normal AddedInterface pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Add eth0 [10.134.0.173/23] from ovn-kubernetes 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Container created 23m Normal Started pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Container started 23m Normal Created pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Container created 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Add eth0 [10.134.0.174/23] from ovn-kubernetes 23m Normal Started pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Container started 23m Normal Pulled pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal AddedLabel gateway/e2e-ipp-b-5deb36 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-5deb36 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw 23m Normal ScalingReplicaSet deployment/e2e-ipp-b-5deb36-openshift-default Scaled up replica set e2e-ipp-b-5deb36-openshift-default-5c58465fc5 from 0 to 1 23m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Add eth0 [10.134.0.176/23] from ovn-kubernetes 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f Created pod: payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-5deb36 Scaled up replica set payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-5deb36-6cc8486645 Created pod: payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-5deb36-6cc8486645 Error creating: pods "payload-processing-e2e-ipp-b-5deb36-6cc8486645-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5deb36: serviceaccount "payload-processing-e2e-ipp-b-5deb36" not found 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-5deb36 Scaled up replica set payload-processing-e2e-ipp-b-5deb36-6cc8486645 from 0 to 1 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f Error creating: pods "payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5deb36: serviceaccount "payload-processing-e2e-ipp-b-5deb36" not found 23m Normal AddedInterface pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Add eth0 [10.133.1.121/23] from ovn-kubernetes 23m Normal Pulled pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Started pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Container started 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Add eth0 [10.133.1.120/23] from ovn-kubernetes 23m Normal Created pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Container created 23m Normal Started pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Container started 23m Normal Created pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Container created 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-1354a5-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 23m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Stopping container istio-proxy 23m Normal SuccessfulDelete replicaset/e2e-worker-w6-1354a5-openshift-default-6c46966c87 Deleted pod: e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d 23m Normal ScalingReplicaSet deployment/e2e-worker-w6-1354a5-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-1354a5-openshift-default-6c46966c87 from 3 to 2 23m Normal Killing pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 Stopping container istio-proxy 23m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 Deleted pod: e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 23m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cfe14f-openshift-default Scaled down replica set e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457 from 4 to 3 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cfe14f-openshift-default New size: 3; reason: All metrics below target 23m Warning Unhealthy pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-5cb2d Readiness probe error: HTTP probe failed with statuscode: 503... 23m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-03d349-openshift-default-cdb65584c Deleted pod: e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 23m Normal Created pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Container created 23m Normal Pulled pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Started pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Container started 23m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-03d349-openshift-default Scaled down replica set e2e-shared-b-w0-03d349-openshift-default-cdb65584c from 3 to 2 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-03d349-openshift-default New size: 2; reason: All metrics below target 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-f7ad21-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-f7ad21-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Normal Killing pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 Stopping container istio-proxy 23m Normal Killing pod/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6 Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969 Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-ipp-a-db0540-85875d5ff6-hgxhj Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-ipp-a-db0540-786d5c8cd7-rzbxj Stopping container payload-pre-processing 23m Warning ProbeError pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Startup probe error: Get "http://10.134.0.169:15021/healthz/ready": dial tcp 10.134.0.169:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Startup probe failed: Get "http://10.134.0.169:15021/healthz/ready": dial tcp 10.134.0.169:15021: connect: connection refused 23m Normal Killing pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Stopping container istio-proxy 23m Warning Unhealthy pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-f7ad21-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-f7ad21-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Normal Killing pod/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2 Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9 Stopping container payload-processing 22m Normal Killing pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw Stopping container istio-proxy 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-f7ad21 Scaled up replica set payload-processing-e2e-ipp-b-f7ad21-c576654b6 from 0 to 1 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-f7ad21-c576654b6 Created pod: payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-f7ad21 Scaled up replica set payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff Error creating: pods "payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f7ad21: serviceaccount "payload-processing-e2e-ipp-b-f7ad21" not found 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-f7ad21-c576654b6 Error creating: pods "payload-processing-e2e-ipp-b-f7ad21-c576654b6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f7ad21: serviceaccount "payload-processing-e2e-ipp-b-f7ad21" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff Created pod: payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 22m Normal Created pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Container created 22m Normal Started pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Container started 22m Normal AddedInterface pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Add eth0 [10.133.1.125/23] from ovn-kubernetes 22m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 22m Normal Pulled pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Add eth0 [10.133.1.124/23] from ovn-kubernetes 22m Normal Created pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Container created 22m Normal Started pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Container started 22m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Stopping container payload-processing 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 22m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-0f980e-openshift-default Scaled up replica set e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 from 3 to 4 22m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d 22m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Container created 22m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Add eth0 [10.134.0.179/23] from ovn-kubernetes 22m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Container started 22m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf Created pod: e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd 22m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-d1bfce-openshift-default Scaled up replica set e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf from 3 to 4 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-d1bfce-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 22m Normal Created pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Container created 22m Normal AddedInterface pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Add eth0 [10.134.0.180/23] from ovn-kubernetes 22m Normal Pulled pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Started pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Container started 22m Warning Unhealthy pod/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 22m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Startup probe failed: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused 22m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 22m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Startup probe error: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Startup probe failed: Get "http://10.134.0.180:15021/healthz/ready": dial tcp 10.134.0.180:15021: connect: connection refused 22m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Startup probe error: Get "http://10.134.0.180:15021/healthz/ready": dial tcp 10.134.0.180:15021: connect: connection refused... 21m Normal Killing pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Stopping container payload-processing 21m Normal Killing pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx Stopping container istio-proxy 21m Normal Killing pod/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p Stopping container payload-pre-processing 21m Normal Killing pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Stopping container istio-proxy 21m Normal Killing pod/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn Stopping container payload-processing 21m Normal Killing pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-vqkgr Stopping container istio-proxy 21m Warning Unhealthy pod/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4 Readiness probe failed: timeout: failed to connect service "10.134.0.174:50052" within 1s: context deadline exceeded 21m Warning ProbeError pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning Unhealthy pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning Unhealthy pod/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2 Readiness probe failed: timeout: failed to connect service "10.134.0.173:50052" within 1s: context deadline exceeded 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv 21m Normal ScalingReplicaSet deployment/e2e-ipp-b-5deb36-openshift-default Scaled up replica set e2e-ipp-b-5deb36-openshift-default-5c58465fc5 from 1 to 4 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc 21m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Add eth0 [10.133.1.130/23] from ovn-kubernetes 21m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Container started 21m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Add eth0 [10.134.0.181/23] from ovn-kubernetes 21m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Add eth0 [10.132.0.174/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Container created 21m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Container created 21m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Container started 21m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-25d881-openshift-default Scaled up replica set e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7 from 1 to 2 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w2-25d881-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 21m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7 Created pod: e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 21m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Container started 21m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Readiness probe failed: Get "http://10.134.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Readiness probe error: Get "http://10.134.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Normal AddedInterface pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Add eth0 [10.133.1.131/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c 21m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Container created 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 21m Normal Created pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Container created 21m Normal ScalingReplicaSet deployment/e2e-ipp-b-d51cd8-openshift-default Scaled up replica set e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c from 1 to 4 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm 21m Normal Started pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Container started 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb 21m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Container started 21m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Container started 21m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Container created 21m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Container created 21m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Add eth0 [10.134.0.182/23] from ovn-kubernetes 21m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Add eth0 [10.132.0.175/23] from ovn-kubernetes 21m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Add eth0 [10.133.1.132/23] from ovn-kubernetes 21m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Container created 21m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Container started 21m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Readiness probe failed: Get "http://10.134.0.172:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Readiness probe error: Get "http://10.134.0.172:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Startup probe error: Get "http://10.132.0.174:15021/healthz/ready": dial tcp 10.132.0.174:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Startup probe failed: Get "http://10.132.0.174:15021/healthz/ready": dial tcp 10.132.0.174:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Startup probe error: Get "http://10.133.1.130:15021/healthz/ready": dial tcp 10.133.1.130:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Startup probe failed: Get "http://10.133.1.130:15021/healthz/ready": dial tcp 10.133.1.130:15021: connect: connection refused 21m Normal Killing pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Stopping container payload-pre-processing 21m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Startup probe failed: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused 21m Warning ProbeError pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Startup probe error: Get "http://10.133.1.131:15021/healthz/ready": dial tcp 10.133.1.131:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Startup probe failed: Get "http://10.133.1.131:15021/healthz/ready": dial tcp 10.133.1.131:15021: connect: connection refused 21m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Container started 21m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Container created 21m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Startup probe error: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused... 21m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Startup probe error: Get "http://10.133.1.132:15021/healthz/ready": dial tcp 10.133.1.132:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Startup probe failed: Get "http://10.133.1.132:15021/healthz/ready": dial tcp 10.133.1.132:15021: connect: connection refused 21m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Startup probe failed: Get "http://10.132.0.175:15021/healthz/ready": dial tcp 10.132.0.175:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Startup probe error: Get "http://10.132.0.175:15021/healthz/ready": dial tcp 10.132.0.175:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Startup probe failed: Get "http://10.134.0.182:15021/healthz/ready": dial tcp 10.134.0.182:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Startup probe error: Get "http://10.134.0.182:15021/healthz/ready": dial tcp 10.134.0.182:15021: connect: connection refused... 21m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Stopping container payload-processing 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Container created 21m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Startup probe error: Get "http://10.134.0.176:15021/healthz/ready": dial tcp 10.134.0.176:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Startup probe failed: Get "http://10.134.0.176:15021/healthz/ready": dial tcp 10.134.0.176:15021: connect: connection refused 21m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Container started 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Startup probe failed: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Startup probe error: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused... 21m Warning Unhealthy pod/payload-processing-e2e-shared-b-w0-03d349-77c695ffdf-dfjbm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w0-03d349-6747675b67-qzzqt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal SuccessfulDelete replicaset/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c Deleted pod: e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz 21m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Stopping container istio-proxy 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-4a68a1-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-worker-w1-4a68a1-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-4a68a1-openshift-default-64ff97f89c from 3 to 2 21m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-8mgrz Readiness probe error: HTTP probe failed with statuscode: 503... 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-477f6b-openshift-default New size: 1; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-477f6b-openshift-default Scaled down replica set e2e-shared-a-w6-477f6b-openshift-default-7854c78755 from 2 to 1 21m Normal Killing pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 Stopping container istio-proxy 21m Normal SuccessfulDelete replicaset/e2e-shared-a-w6-477f6b-openshift-default-7854c78755 Deleted pod: e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-6b5b1e-openshift-default New size: 2; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-6b5b1e-openshift-default Scaled down replica set e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 from 3 to 2 21m Normal Killing pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb Stopping container istio-proxy 21m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f Deleted pod: e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb 21m Normal SuccessfulDelete replicaset/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695 Deleted pod: e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-f22a52-openshift-default New size: 1; reason: All metrics below target 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-f22a52-openshift-default Scaled down replica set e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f from 2 to 1 21m Normal Killing pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj Stopping container istio-proxy 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal SuccessfulDelete replicaset/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 Deleted pod: e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv 21m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-c73fcb-openshift-default Scaled down replica set e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55 from 3 to 2 21m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv Stopping container istio-proxy 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-c73fcb-openshift-default New size: 2; reason: All metrics below target 21m Normal Killing pod/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg Stopping container payload-processing 21m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-d51cd8 Scaled up replica set payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679 from 0 to 1 21m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679 Error creating: pods "payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-d51cd8: serviceaccount "payload-processing-e2e-ipp-b-d51cd8" not found 21m Normal Killing pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Stopping container istio-proxy 21m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58847f9fc5 from 4 to 3 21m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58847f9fc5 Deleted pod: maas-default-gateway-openshift-default-58847f9fc5-7dz5z 21m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-d51cd8-6697d94d98 Error creating: pods "payload-processing-e2e-ipp-b-d51cd8-6697d94d98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-d51cd8: serviceaccount "payload-processing-e2e-ipp-b-d51cd8" not found 21m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-d51cd8 Scaled up replica set payload-processing-e2e-ipp-b-d51cd8-6697d94d98 from 0 to 1 21m Normal AddedInterface pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Add eth0 [10.132.0.176/23] from ovn-kubernetes 21m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679 Created pod: payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 21m Normal Pulled pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 21m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-d51cd8-6697d94d98 Created pod: payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 21m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Add eth0 [10.133.1.134/23] from ovn-kubernetes 21m Normal Created pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Container created 21m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 21m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Created pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Container created 21m Normal Started pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Container started 21m Normal Started pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Container started 21m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z Readiness probe error: HTTP probe failed with statuscode: 503... 20m Normal Killing pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 Stopping container istio-proxy 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n 20m Normal ScalingReplicaSet deployment/e2e-ipp-b-5deb36-openshift-default Scaled up replica set e2e-ipp-b-5deb36-openshift-default-5c58465fc5 from 4 to 8 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default New size: 8; reason: All metrics below target 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Container started 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Add eth0 [10.134.0.183/23] from ovn-kubernetes 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Container started 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Container created 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Container created 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Container started 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Add eth0 [10.133.1.136/23] from ovn-kubernetes 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Container started 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Add eth0 [10.132.0.177/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Container created 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Add eth0 [10.133.1.137/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Container created 20m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-5deb36 Scaled up replica set payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc from 0 to 1 20m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc Error creating: pods "payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5deb36: serviceaccount "payload-processing-e2e-ipp-b-5deb36" not found 20m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-5deb36 Scaled up replica set payload-processing-e2e-ipp-b-5deb36-5b9855cb7d from 0 to 1 20m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d Error creating: pods "payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5deb36: serviceaccount "payload-processing-e2e-ipp-b-5deb36" not found 20m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d Created pod: payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg 20m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc Created pod: payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 20m Normal Started pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Container started 20m Normal Created pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Container created 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default New size: 8; reason: All metrics below target 20m Normal ScalingReplicaSet deployment/e2e-ipp-b-d51cd8-openshift-default Scaled up replica set e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c from 4 to 8 20m Normal AddedInterface pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Add eth0 [10.132.0.178/23] from ovn-kubernetes 20m Normal Pulled pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Add eth0 [10.133.1.140/23] from ovn-kubernetes 20m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Add eth0 [10.134.0.184/23] from ovn-kubernetes 20m Normal Created pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Container created 20m Normal Started pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Container started 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Container started 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Container created 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Add eth0 [10.132.0.179/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Container created 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Add eth0 [10.133.1.139/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Add eth0 [10.133.1.138/23] from ovn-kubernetes 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Container started 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Container started 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Container created 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Container started 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Container created 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Startup probe failed: Get "http://10.133.1.136:15021/healthz/ready": dial tcp 10.133.1.136:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Startup probe error: Get "http://10.133.1.136:15021/healthz/ready": dial tcp 10.133.1.136:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Startup probe error: Get "http://10.132.0.177:15021/healthz/ready": dial tcp 10.132.0.177:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Startup probe failed: Get "http://10.134.0.183:15021/healthz/ready": dial tcp 10.134.0.183:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Startup probe error: Get "http://10.134.0.183:15021/healthz/ready": dial tcp 10.134.0.183:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Startup probe failed: Get "http://10.132.0.177:15021/healthz/ready": dial tcp 10.132.0.177:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Startup probe failed: Get "http://10.133.1.137:15021/healthz/ready": dial tcp 10.133.1.137:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Startup probe error: Get "http://10.133.1.137:15021/healthz/ready": dial tcp 10.133.1.137:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Startup probe failed: Get "http://10.134.0.184:15021/healthz/ready": dial tcp 10.134.0.184:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Startup probe failed: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Startup probe error: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Startup probe error: Get "http://10.134.0.184:15021/healthz/ready": dial tcp 10.134.0.184:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Startup probe error: Get "http://10.133.1.140:15021/healthz/ready": dial tcp 10.133.1.140:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Startup probe failed: Get "http://10.133.1.140:15021/healthz/ready": dial tcp 10.133.1.140:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Startup probe error: Get "http://10.133.1.139:15021/healthz/ready": dial tcp 10.133.1.139:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Startup probe failed: Get "http://10.133.1.139:15021/healthz/ready": dial tcp 10.133.1.139:15021: connect: connection refused 20m Normal Killing pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw Stopping container istio-proxy 20m Normal Killing pod/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp Stopping container payload-processing 20m Normal Killing pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q9hsw Stopping container istio-proxy 20m Normal Killing pod/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6 Stopping container payload-pre-processing 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 (combined from similar events): Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-5deb36-openshift-default-5c58465fc5 Created pod: e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd 20m Normal ScalingReplicaSet deployment/e2e-ipp-b-5deb36-openshift-default Scaled up replica set e2e-ipp-b-5deb36-openshift-default-5c58465fc5 from 8 to 10 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-5deb36-openshift-default New size: 10; reason: All metrics below target 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Add eth0 [10.134.0.185/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Container created 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Container started 20m Normal AddedInterface pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Add eth0 [10.132.0.180/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Container created 20m Normal Started pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Container started 20m Normal ScalingReplicaSet deployment/e2e-ipp-b-d51cd8-openshift-default Scaled up replica set e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c from 8 to 10 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-d51cd8-openshift-default New size: 10; reason: 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr 20m Normal SuccessfulCreate replicaset/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c (combined from similar events): Created pod: e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Container created 20m Normal Created pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Container created 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Container started 20m Normal Started pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Container started 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Add eth0 [10.132.0.181/23] from ovn-kubernetes 20m Normal AddedInterface pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Add eth0 [10.134.0.187/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Pulled pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Startup probe error: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Startup probe failed: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Startup probe failed: Get "http://10.132.0.180:15021/healthz/ready": dial tcp 10.132.0.180:15021: connect: connection refused 20m Normal Killing pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t Stopping container istio-proxy 20m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Startup probe error: Get "http://10.132.0.180:15021/healthz/ready": dial tcp 10.132.0.180:15021: connect: connection refused... 20m Normal Killing pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 Stopping container istio-proxy 20m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-pr4wx Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Startup probe failed: Get "http://10.132.0.181:15021/healthz/ready": dial tcp 10.132.0.181:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Startup probe failed: Get "http://10.134.0.187:15021/healthz/ready": dial tcp 10.134.0.187:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Startup probe error: Get "http://10.134.0.187:15021/healthz/ready": dial tcp 10.134.0.187:15021: connect: connection refused... 20m Normal Killing pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Stopping container payload-pre-processing 20m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Startup probe error: Get "http://10.132.0.181:15021/healthz/ready": dial tcp 10.132.0.181:15021: connect: connection refused... 20m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2 Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Stopping container payload-processing 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-cc33d3-openshift-default New size: 2; reason: All metrics below target 20m Normal SuccessfulDelete replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Deleted pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 20m Normal SuccessfulDelete replicaset/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 Deleted pod: e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs 20m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-cc33d3-openshift-default Scaled down replica set e2e-shared-b-w4-cc33d3-openshift-default-6d67448675 from 4 to 2 20m Normal Killing pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs Stopping container istio-proxy 20m Normal Killing pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx Stopping container istio-proxy 20m Warning Unhealthy pod/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg Stopping container istio-proxy 19m Normal Killing pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Stopping container payload-pre-processing 19m Normal Killing pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l Stopping container istio-proxy 19m Warning ProbeError pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning Unhealthy pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq Readiness probe failed: HTTP probe failed with statuscode: 503 19m Normal Killing pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Stopping container payload-pre-processing 19m Normal Killing pod/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-ipp-b-5deb36-5b9855cb7d-wbhsg Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj Stopping container payload-processing 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-t6pjc Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-xqqxd Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Stopping container istio-proxy 19m Warning Unhealthy pod/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-49f6n Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn Stopping container istio-proxy 19m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning ProbeError pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv Readiness probe failed: HTTP probe failed with statuscode: 503 19m Normal ScalingReplicaSet deployment/e2e-ipp-a-1685dd-openshift-default Scaled up replica set e2e-ipp-a-1685dd-openshift-default-79d84f767 from 0 to 1 19m Normal AddedLabel gateway/e2e-ipp-a-1685dd Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-1685dd 19m Normal SuccessfulCreate replicaset/e2e-ipp-a-1685dd-openshift-default-79d84f767 Created pod: e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 19m Warning Unhealthy pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal AddedInterface pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Add eth0 [10.133.1.157/23] from ovn-kubernetes 19m Normal Killing pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc Stopping container istio-proxy 19m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-0f980e-openshift-default Scaled up replica set e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 from 4 to 6 19m Normal Killing pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-sr986 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l Stopping container istio-proxy 19m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 19m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-0f980e-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 19m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67 Created pod: e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 19m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Add eth0 [10.133.1.160/23] from ovn-kubernetes 19m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Container created 19m Normal Created pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Container created 19m Normal AddedInterface pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Add eth0 [10.132.0.182/23] from ovn-kubernetes 19m Warning Unhealthy pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Pulled pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Container started 19m Normal Started pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Container started 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-1685dd Scaled up replica set payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c Created pod: payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-1685dd Scaled up replica set payload-processing-e2e-ipp-a-1685dd-5678c7df5c from 0 to 1 19m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-1685dd-5678c7df5c Error creating: pods "payload-processing-e2e-ipp-a-1685dd-5678c7df5c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1685dd: serviceaccount "payload-processing-e2e-ipp-a-1685dd" not found 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-1685dd-5678c7df5c Created pod: payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c Error creating: pods "payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1685dd: serviceaccount "payload-processing-e2e-ipp-a-1685dd" not found 19m Normal Pulled pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal AddedInterface pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Add eth0 [10.133.1.163/23] from ovn-kubernetes 19m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Add eth0 [10.134.0.189/23] from ovn-kubernetes 19m Normal Created pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Container created 19m Normal Started pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Container started 19m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal Started pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Container started 19m Normal Created pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Container created 19m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Stopping container payload-processing 19m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Stopping container istio-proxy 19m Warning Unhealthy pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Startup probe failed: Get "http://10.133.1.160:15021/healthz/ready": dial tcp 10.133.1.160:15021: connect: connection refused 19m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Startup probe error: Get "http://10.133.1.160:15021/healthz/ready": dial tcp 10.133.1.160:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Startup probe failed: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused 19m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Startup probe error: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused... 19m Warning ProbeError pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lxfj8 Readiness probe error: HTTP probe failed with statuscode: 503... 19m Normal Killing pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Stopping container payload-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Stopping container payload-pre-processing 19m Warning Unhealthy pod/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h Readiness probe failed: timeout: failed to connect service "10.133.0.53:50052" within 1s: context deadline exceeded 18m Warning Unhealthy pod/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Normal Killing pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv Stopping container istio-proxy 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57 Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn Stopping container payload-processing 18m Warning Unhealthy pod/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-fgrq2 Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Stopping container istio-proxy 18m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz Stopping container istio-proxy 18m Normal Killing pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-fk8h9 Stopping container istio-proxy 18m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-mn94c Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning Unhealthy pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal Killing pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Stopping container payload-pre-processing 18m Warning Unhealthy pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-8l82j Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal Killing pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Stopping container payload-processing 18m Normal Killing pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Stopping container payload-pre-processing 18m Normal Started pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Container started 18m Normal Pulled pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Created pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Container created 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-1685dd-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-1685dd-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning Unhealthy pod/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx Stopping container payload-processing 18m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Startup probe failed: Get "http://10.133.1.157:15021/healthz/ready": dial tcp 10.133.1.157:15021: connect: connection refused 18m Warning ProbeError pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Startup probe error: Get "http://10.133.1.157:15021/healthz/ready": dial tcp 10.133.1.157:15021: connect: connection refused... 18m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-7wq2x Stopping container istio-proxy 18m Warning Unhealthy pod/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-1685dd-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-1685dd-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7 Created pod: payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm 18m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db Error creating: pods "payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1685dd: serviceaccount "payload-processing-e2e-ipp-a-1685dd" not found 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db Created pod: payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-1685dd Scaled up replica set payload-processing-e2e-ipp-a-1685dd-7dd48cb8db from 0 to 1 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-1685dd Scaled up replica set payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7 Error creating: pods "payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1685dd: serviceaccount "payload-processing-e2e-ipp-a-1685dd" not found 17m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Add eth0 [10.133.1.173/23] from ovn-kubernetes 17m Normal Started pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Container started 17m Normal Killing pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Stopping container payload-pre-processing 17m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Normal Created pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Container created 17m Normal Killing pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Stopping container payload-processing 17m Normal Started pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Container started 17m Normal AddedInterface pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Add eth0 [10.133.1.174/23] from ovn-kubernetes 17m Normal Pulled pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Normal Created pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Container created 17m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp Stopping container istio-proxy 17m Normal Killing pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn Stopping container istio-proxy 17m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Stopping container istio-proxy 17m Normal Killing pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt Stopping container istio-proxy 17m Warning Unhealthy pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz Readiness probe error: HTTP probe failed with statuscode: 503... 17m Normal Killing pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm Stopping container istio-proxy 17m Normal ScalingReplicaSet deployment/e2e-ipp-b-630fdf-openshift-default Scaled up replica set e2e-ipp-b-630fdf-openshift-default-7d5dd78ff from 0 to 1 17m Normal AddedLabel gateway/e2e-ipp-b-630fdf Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-630fdf 17m Normal SuccessfulCreate replicaset/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff Created pod: e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs 17m Normal AddedInterface pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Add eth0 [10.134.0.190/23] from ovn-kubernetes 17m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-630fdf-5fdf864f85 Error creating: pods "payload-processing-e2e-ipp-b-630fdf-5fdf864f85-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-630fdf: serviceaccount "payload-processing-e2e-ipp-b-630fdf" not found 17m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-630fdf Scaled up replica set payload-processing-e2e-ipp-b-630fdf-5fdf864f85 from 0 to 1 17m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f Created pod: payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 17m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-630fdf Scaled up replica set payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f from 0 to 1 17m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-630fdf-5fdf864f85 Created pod: payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt 17m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f Error creating: pods "payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-630fdf: serviceaccount "payload-processing-e2e-ipp-b-630fdf" not found 17m Normal Created pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Container created 17m Normal Started pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Container started 17m Normal Started pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Container started 17m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Add eth0 [10.134.0.193/23] from ovn-kubernetes 17m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 17m Normal Pulled pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 17m Normal Created pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Container created 17m Normal AddedInterface pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Add eth0 [10.134.0.192/23] from ovn-kubernetes 17m Warning Unhealthy pod/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Killing pod/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26 Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-ipp-b-d51cd8-6697d94d98-whg99 Stopping container payload-processing 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-vr7zr Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9l7mb Stopping container istio-proxy 17m Normal Killing pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf Stopping container istio-proxy 17m Warning FailedToRetrieveImagePullSecret pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-9fn6n Unable to retrieve some image pull secrets (e2e-ipp-b-d51cd8-openshift-default-dockercfg-gdp6w); attempting to pull the image may not succeed. 17m Warning Unhealthy pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Readiness probe failed: HTTP probe failed with statuscode: 503 17m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm Stopping container istio-proxy 17m Warning ProbeError pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf Readiness probe error: HTTP probe failed with statuscode: 503... 17m Normal Killing pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-ttzks Stopping container istio-proxy 17m Normal Killing pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Stopping container payload-pre-processing 17m Normal Created pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Container created 17m Normal Started pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Container started 17m Normal Pulled pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 16m Normal Killing pod/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5 Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt Stopping container payload-processing 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-630fdf-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 16m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-630fdf-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 16m Warning Unhealthy pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Startup probe failed: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused 16m Warning ProbeError pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Startup probe error: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused... 16m Warning Unhealthy pod/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf Stopping container istio-proxy 16m Warning ProbeError pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Killing pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r Stopping container istio-proxy 16m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79 Created pod: payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 16m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-630fdf-75fd9cc78 Error creating: pods "payload-processing-e2e-ipp-b-630fdf-75fd9cc78-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-630fdf: serviceaccount "payload-processing-e2e-ipp-b-630fdf" not found 16m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-630fdf Scaled up replica set payload-processing-e2e-ipp-b-630fdf-75fd9cc78 from 0 to 1 16m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-630fdf-75fd9cc78 Created pod: payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 16m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-630fdf Scaled up replica set payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79 from 0 to 1 16m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79 Error creating: pods "payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-630fdf: serviceaccount "payload-processing-e2e-ipp-b-630fdf" not found 16m Normal Pulling pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 16m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Add eth0 [10.134.0.194/23] from ovn-kubernetes 16m Normal AddedInterface pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Add eth0 [10.134.0.195/23] from ovn-kubernetes 16m Normal Pulling pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 16m Normal Created pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Container created 16m Normal Started pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Container started 16m Normal Started pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Container started 16m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.982s (1.982s including waiting). Image size: 166612641 bytes. 16m Normal Created pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Container created 16m Normal Pulled pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.817s (1.817s including waiting). Image size: 166612641 bytes. 16m Normal Killing pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Stopping container payload-processing 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-630fdf-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-630fdf-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning Unhealthy pod/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Stopping container istio-proxy 16m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-kk6sx Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Killing pod/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Stopping container payload-pre-processing 16m Normal Killing pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Stopping container istio-proxy 15m Warning Unhealthy pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6 Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2 Stopping container payload-pre-processing 15m Warning Unhealthy pod/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Normal Killing pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs Stopping container istio-proxy 15m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Stopping container istio-proxy 15m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Stopping container istio-proxy 15m Warning ProbeError pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-hnpsw Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning Unhealthy pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 Readiness probe failed: HTTP probe failed with statuscode: 503 15m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Stopping container payload-pre-processing 15m Normal Killing pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Stopping container payload-processing 15m Normal Killing pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 Stopping container istio-proxy 15m Warning ProbeError pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning Unhealthy pod/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Normal Killing pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Stopping container payload-pre-processing 14m Normal Killing pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Stopping container istio-proxy 14m Warning Unhealthy pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning ProbeError pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn Readiness probe error: HTTP probe failed with statuscode: 503... 14m Normal Killing pod/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6 Stopping container payload-processing 14m Normal Killing pod/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp Stopping container payload-pre-processing 14m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 Stopping container istio-proxy 14m Normal Killing pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mlt2t Stopping container istio-proxy 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-w8wfj 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-n57mc 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-qgv7r 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Add eth0 [10.134.0.196/23] from ovn-kubernetes 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Add eth0 [10.132.0.186/23] from ovn-kubernetes 14m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 3 to 6 14m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Add eth0 [10.132.0.185/23] from ovn-kubernetes 14m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Container created 14m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 14m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Container started 14m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Container created 14m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 14m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Container started 14m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Container created 14m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 14m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Container started 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Startup probe error: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Startup probe failed: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Startup probe failed: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Startup probe error: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused... 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Startup probe error: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Startup probe failed: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Add eth0 [10.133.1.190/23] from ovn-kubernetes 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 6 to 7 13m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-nrm2h 13m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Container created 13m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Container started 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Startup probe error: Get "http://10.133.1.190:15021/healthz/ready": dial tcp 10.133.1.190:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Startup probe failed: Get "http://10.133.1.190:15021/healthz/ready": dial tcp 10.133.1.190:15021: connect: connection refused 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-5cp4f 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-74tmp 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58847f9fc5 from 7 to 10 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58847f9fc5 Created pod: maas-default-gateway-openshift-default-58847f9fc5-mbq7p 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 12m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Container created 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Add eth0 [10.132.0.187/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Container created 12m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Container started 12m Normal Created pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Container created 12m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Container started 12m Normal Started pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Add eth0 [10.133.1.193/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Add eth0 [10.134.0.197/23] from ovn-kubernetes 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Startup probe error: Get "http://10.132.0.187:15021/healthz/ready": dial tcp 10.132.0.187:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Startup probe failed: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Startup probe failed: Get "http://10.132.0.187:15021/healthz/ready": dial tcp 10.132.0.187:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Startup probe error: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Startup probe error: Get "http://10.133.1.193:15021/healthz/ready": dial tcp 10.133.1.193:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Startup probe failed: Get "http://10.133.1.193:15021/healthz/ready": dial tcp 10.133.1.193:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Startup probe error: Get "http://10.132.0.187:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-5cp4f Startup probe failed: Get "http://10.132.0.187:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Startup probe failed: Get "http://10.134.0.197:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp Startup probe error: Get "http://10.134.0.197:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 12m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 12m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-d54kn 12m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-d54kn Stopping container discovery 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Startup probe error: Get "http://10.133.1.193:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p Startup probe failed: Get "http://10.133.1.193:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 9m3s Normal AddedInterface pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Add eth0 [10.134.0.198/23] from ovn-kubernetes 9m3s Normal SuccessfulCreate replicaset/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7 Created pod: e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 9m3s Normal AddedLabel gateway/e2e-worker-single-main-6c0c09 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-6c0c09 9m3s Normal ScalingReplicaSet deployment/e2e-worker-single-main-6c0c09-openshift-default Scaled up replica set e2e-worker-single-main-6c0c09-openshift-default-5c656554c7 from 0 to 1 8m58s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c6664d7 Error creating: pods "payload-pre-processing-e2e-worker-single-main-6c0c09-d8c6664d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09: serviceaccount "payload-processing-e2e-worker-single-main-6c0c09" not found 8m58s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-6c0c09 Scaled up replica set payload-processing-e2e-worker-single-main-6c0c09-6797c446b9 from 0 to 1 8m58s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-6c0c09-6797c446b9 Created pod: payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz 8m58s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-6c0c09-6797c446b9 Error creating: pods "payload-processing-e2e-worker-single-main-6c0c09-6797c446b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09: serviceaccount "payload-processing-e2e-worker-single-main-6c0c09" not found 8m58s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-6c0c09 Scaled up replica set payload-pre-processing-e2e-worker-single-main-6c0c09-d8c6664d7 from 0 to 1 8m58s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c6664d7 Created pod: payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 8m57s Normal Pulled pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 8m57s Normal Created pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Container created 8m57s Normal Started pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Container started 8m57s Normal Started pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Container started 8m57s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Add eth0 [10.133.1.206/23] from ovn-kubernetes 8m57s Normal Created pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Container created 8m57s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 8m57s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Add eth0 [10.134.0.199/23] from ovn-kubernetes 8m45s Warning ProbeError pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Readiness probe error: Get "http://10.134.0.198:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 8m45s Warning Unhealthy pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Readiness probe failed: Get "http://10.134.0.198:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 8m44s Normal Created pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Container created 8m44s Normal Pulled pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m44s Normal Started pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Container started 8m39s Warning Unhealthy pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Startup probe failed: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused 8m39s Warning ProbeError pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Startup probe error: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused... 8m3s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-6c0c09-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m3s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-6c0c09-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m57s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Stopping container payload-pre-processing 7m57s Normal Killing pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Stopping container payload-processing 7m48s Normal Killing pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 Stopping container istio-proxy 7m36s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m35s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m59s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Readiness probe failed: Get "http://10.133.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m59s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 Readiness probe error: Get "http://10.133.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m30s Normal AddedLabel gateway/e2e-worker-single-main-ee1c63 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-ee1c63 5m30s Normal ScalingReplicaSet deployment/e2e-worker-single-main-ee1c63-openshift-default Scaled up replica set e2e-worker-single-main-ee1c63-openshift-default-579d85c845 from 0 to 1 5m30s Normal AddedInterface pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Add eth0 [10.134.0.201/23] from ovn-kubernetes 5m30s Normal SuccessfulCreate replicaset/e2e-worker-single-main-ee1c63-openshift-default-579d85c845 Created pod: e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq 5m25s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-ee1c63-7f98864685 Error creating: pods "payload-processing-e2e-worker-single-main-ee1c63-7f98864685-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63: serviceaccount "payload-processing-e2e-worker-single-main-ee1c63" not found 5m25s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6df44c9 Created pod: payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 5m25s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-ee1c63-7f98864685 Created pod: payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c 5m25s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-ee1c63 Scaled up replica set payload-pre-processing-e2e-worker-single-main-ee1c63-7df6df44c9 from 0 to 1 5m25s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-ee1c63 Scaled up replica set payload-processing-e2e-worker-single-main-ee1c63-7f98864685 from 0 to 1 5m25s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6df44c9 Error creating: pods "payload-pre-processing-e2e-worker-single-main-ee1c63-7df6df44c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63: serviceaccount "payload-processing-e2e-worker-single-main-ee1c63" not found 5m24s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Add eth0 [10.134.0.202/23] from ovn-kubernetes 5m24s Normal Started pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Container started 5m24s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m24s Normal Created pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Container created 5m24s Normal Started pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Container started 5m24s Normal Pulled pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m24s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Add eth0 [10.133.1.211/23] from ovn-kubernetes 5m24s Normal Created pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Container created 5m12s Normal Started pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Container started 5m12s Normal Pulled pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m12s Warning Unhealthy pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Readiness probe failed: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused 5m12s Warning ProbeError pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Readiness probe error: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused... 5m12s Normal Created pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Container created 5m7s Warning Unhealthy pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Startup probe failed: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused 5m7s Warning ProbeError pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Startup probe error: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused... 4m45s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ee1c63-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m45s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ee1c63-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m29s Normal Killing pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Stopping container payload-processing 4m29s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Stopping container payload-pre-processing 4m19s Normal Killing pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq Stopping container istio-proxy 4m13s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m13s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 3m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Readiness probe failed: Get "http://10.134.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m52s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Readiness probe error: Get "http://10.133.1.190:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h Readiness probe failed: Get "http://10.133.1.190:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m52s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj Readiness probe error: Get "http://10.134.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m37s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Readiness probe error: Get "http://10.132.0.186:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m37s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-qgv7r Readiness probe failed: Get "http://10.132.0.186:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m37s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Readiness probe error: Get "http://10.132.0.185:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m37s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-n57mc Readiness probe failed: Get "http://10.132.0.185:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m12s Warning Unhealthy pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Readiness probe failed: Get "http://10.134.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m12s Warning ProbeError pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh Readiness probe error: Get "http://10.134.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...