=== events in openshift-ingress (2026-10-30T18:53:02Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 29m Normal Scheduled pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Successfully assigned openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Successfully assigned openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r to ip-10-0-143-94.ec2.internal 37m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd to ip-10-0-143-94.ec2.internal 40m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv to ip-10-0-143-94.ec2.internal 36m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 to ip-10-0-143-94.ec2.internal 41m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b to ip-10-0-141-106.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Successfully assigned openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z to ip-10-0-143-94.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Successfully assigned openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Successfully assigned openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm to ip-10-0-133-47.ec2.internal 24m Normal Scheduled pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Successfully assigned openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm to ip-10-0-141-106.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Successfully assigned openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 to ip-10-0-133-47.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59 to ip-10-0-133-47.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Successfully assigned openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr to ip-10-0-133-47.ec2.internal 45m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-hn66q Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q to ip-10-0-143-94.ec2.internal 22m Normal Scheduled pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Successfully assigned openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw to ip-10-0-141-106.ec2.internal 52m Normal Scheduled pod/router-default-7765667bbf-4zpgd Successfully assigned openshift-ingress/router-default-7765667bbf-4zpgd to ip-10-0-143-94.ec2.internal 52m Warning FailedScheduling pod/router-default-7765667bbf-4zpgd no nodes available to schedule pods 34m Normal Scheduled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Successfully assigned openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Successfully assigned openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Successfully assigned openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw to ip-10-0-141-106.ec2.internal 5m24s Normal Scheduled pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Successfully assigned openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd to ip-10-0-141-106.ec2.internal 31m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt to ip-10-0-133-47.ec2.internal 38m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc to ip-10-0-133-47.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn to ip-10-0-133-47.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c to ip-10-0-143-94.ec2.internal 40m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx to ip-10-0-133-47.ec2.internal 47m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd to ip-10-0-133-47.ec2.internal 34m Normal Scheduled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Successfully assigned openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 to ip-10-0-143-94.ec2.internal 39m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q to ip-10-0-143-94.ec2.internal 43m Normal Scheduled pod/payload-processing-5f78687d5b-j6n87 Successfully assigned openshift-ingress/payload-processing-5f78687d5b-j6n87 to ip-10-0-133-47.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 to ip-10-0-133-47.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl to ip-10-0-143-94.ec2.internal 39m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Successfully assigned openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w to ip-10-0-141-106.ec2.internal 37m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 to ip-10-0-133-47.ec2.internal 34m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr to ip-10-0-141-106.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 to ip-10-0-133-47.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Successfully assigned openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq to ip-10-0-143-94.ec2.internal 35m Normal Scheduled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Successfully assigned openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p to ip-10-0-133-47.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Successfully assigned openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z to ip-10-0-143-94.ec2.internal 32m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 to ip-10-0-133-47.ec2.internal 41m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb to ip-10-0-143-94.ec2.internal 39m Normal Scheduled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Successfully assigned openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw to ip-10-0-143-94.ec2.internal 39m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv to ip-10-0-133-47.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d to ip-10-0-143-94.ec2.internal 37m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 to ip-10-0-143-94.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 to ip-10-0-133-47.ec2.internal 24m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w to ip-10-0-141-106.ec2.internal 36m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 to ip-10-0-143-94.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 to ip-10-0-133-47.ec2.internal 39m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k to ip-10-0-143-94.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Successfully assigned openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj to ip-10-0-133-47.ec2.internal 38m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 to ip-10-0-133-47.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d to ip-10-0-133-47.ec2.internal 37m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 to ip-10-0-141-106.ec2.internal 8m38s Normal Scheduled pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Successfully assigned openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff to ip-10-0-133-47.ec2.internal 12m Normal Scheduled pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw to ip-10-0-141-106.ec2.internal 37m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj to ip-10-0-141-106.ec2.internal 40m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d to ip-10-0-143-94.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr to ip-10-0-133-47.ec2.internal 38m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr to ip-10-0-133-47.ec2.internal 8m33s Normal Scheduled pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Successfully assigned openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf to ip-10-0-143-94.ec2.internal 42m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w to ip-10-0-133-47.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb to ip-10-0-141-106.ec2.internal 41m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm to ip-10-0-141-106.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Successfully assigned openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Successfully assigned openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm to ip-10-0-133-47.ec2.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 to ip-10-0-141-106.ec2.internal 8m33s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 to ip-10-0-141-106.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x to ip-10-0-133-47.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 to ip-10-0-143-94.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 to ip-10-0-133-47.ec2.internal 32m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp to ip-10-0-133-47.ec2.internal 32m Normal Scheduled pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd to ip-10-0-133-47.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w to ip-10-0-133-47.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 to ip-10-0-133-47.ec2.internal 29m Normal Scheduled pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Successfully assigned openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Successfully assigned openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb to ip-10-0-133-47.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr to ip-10-0-141-106.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r to ip-10-0-133-47.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Successfully assigned openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv to ip-10-0-133-47.ec2.internal 4m48s Normal Scheduled pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Successfully assigned openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 to ip-10-0-133-47.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 to ip-10-0-133-47.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt to ip-10-0-143-94.ec2.internal 5m31s Normal Scheduled pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Successfully assigned openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 to ip-10-0-133-47.ec2.internal 30m Normal Scheduled pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Successfully assigned openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k to ip-10-0-143-94.ec2.internal 4m48s Normal Scheduled pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Successfully assigned openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Successfully assigned openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj to ip-10-0-141-106.ec2.internal 5m24s Normal Scheduled pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g to ip-10-0-143-94.ec2.internal 39m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk to ip-10-0-141-106.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-7b98bb678c-9w8hc Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc to ip-10-0-141-106.ec2.internal 35m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 to ip-10-0-141-106.ec2.internal 29m Normal Scheduled pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Successfully assigned openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 to ip-10-0-141-106.ec2.internal 37m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf to ip-10-0-143-94.ec2.internal 38m Normal Scheduled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Successfully assigned openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr to ip-10-0-143-94.ec2.internal 37m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 to ip-10-0-133-47.ec2.internal 39m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g to ip-10-0-133-47.ec2.internal 12m Normal Scheduled pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Successfully assigned openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj to ip-10-0-141-106.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d to ip-10-0-143-94.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Successfully assigned openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm to ip-10-0-133-47.ec2.internal 40m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst to ip-10-0-141-106.ec2.internal 48m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 to ip-10-0-141-106.ec2.internal 25m Normal Scheduled pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Successfully assigned openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn to ip-10-0-133-47.ec2.internal 45m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt to ip-10-0-141-106.ec2.internal 23m Normal Scheduled pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Successfully assigned openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 to ip-10-0-133-47.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b to ip-10-0-141-106.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz to ip-10-0-141-106.ec2.internal 24m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb to ip-10-0-143-94.ec2.internal 24m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd to ip-10-0-133-47.ec2.internal 28m Normal Scheduled pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Successfully assigned openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p to ip-10-0-133-47.ec2.internal 35m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 to ip-10-0-143-94.ec2.internal 35m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd to ip-10-0-143-94.ec2.internal 37m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z to ip-10-0-133-47.ec2.internal 38m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 to ip-10-0-133-47.ec2.internal 36m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj to ip-10-0-141-106.ec2.internal 24m Normal Scheduled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Successfully assigned openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf to ip-10-0-133-47.ec2.internal 38m Normal Scheduled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Successfully assigned openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m to ip-10-0-141-106.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k to ip-10-0-141-106.ec2.internal 36m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp to ip-10-0-143-94.ec2.internal 36m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn to ip-10-0-143-94.ec2.internal 36m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg to ip-10-0-133-47.ec2.internal 36m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p to ip-10-0-143-94.ec2.internal 32m Normal Scheduled pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Successfully assigned openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft to ip-10-0-141-106.ec2.internal 36m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Successfully assigned openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 to ip-10-0-143-94.ec2.internal 35m Normal Scheduled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Successfully assigned openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 to ip-10-0-133-47.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 to ip-10-0-141-106.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr to ip-10-0-141-106.ec2.internal 36m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz to ip-10-0-133-47.ec2.internal 36m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr to ip-10-0-133-47.ec2.internal 39m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk to ip-10-0-133-47.ec2.internal 36m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 to ip-10-0-141-106.ec2.internal 37m Normal Scheduled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Successfully assigned openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 to ip-10-0-141-106.ec2.internal 40m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f to ip-10-0-133-47.ec2.internal 43m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Successfully assigned openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld to ip-10-0-133-47.ec2.internal 40m Normal Scheduled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Successfully assigned openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q to ip-10-0-143-94.ec2.internal 38m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 to ip-10-0-141-106.ec2.internal 45m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7 to ip-10-0-133-47.ec2.internal 4m54s Normal Scheduled pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Successfully assigned openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph to ip-10-0-141-106.ec2.internal 59m Warning FailedScheduling pod/router-default-7765667bbf-4zpgd no nodes available to schedule pods 36m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 to ip-10-0-141-106.ec2.internal 38m Normal Scheduled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Successfully assigned openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh to ip-10-0-143-94.ec2.internal 27m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 to ip-10-0-133-47.ec2.internal 48m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-7p9tg Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg to ip-10-0-141-106.ec2.internal 44m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2 to ip-10-0-133-47.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 to ip-10-0-141-106.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt to ip-10-0-141-106.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz to ip-10-0-133-47.ec2.internal 44m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-n789k Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k to ip-10-0-143-94.ec2.internal 26m Normal Scheduled pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Successfully assigned openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 to ip-10-0-141-106.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m to ip-10-0-141-106.ec2.internal 32m Normal Scheduled pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Successfully assigned openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 to ip-10-0-133-47.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq to ip-10-0-143-94.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d to ip-10-0-141-106.ec2.internal 33m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 to ip-10-0-141-106.ec2.internal 37m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq to ip-10-0-133-47.ec2.internal 46m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp to ip-10-0-141-106.ec2.internal 34m Normal Scheduled pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Successfully assigned openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 to ip-10-0-141-106.ec2.internal 31m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr to ip-10-0-133-47.ec2.internal 34m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf to ip-10-0-143-94.ec2.internal 43m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-vftgt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt to ip-10-0-143-94.ec2.internal 35m Normal Scheduled pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Successfully assigned openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg to ip-10-0-141-106.ec2.internal 47m Normal Scheduled pod/kube-auth-proxy-69dd9bcffb-ddqsh Successfully assigned openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 to ip-10-0-141-106.ec2.internal 47m Normal Scheduled pod/kube-auth-proxy-69dd9bcffb-zsxmb Successfully assigned openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb to ip-10-0-143-94.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n to ip-10-0-141-106.ec2.internal 44m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb to ip-10-0-141-106.ec2.internal 28m Normal Scheduled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Successfully assigned openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf to ip-10-0-143-94.ec2.internal 43m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-4bknm Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm to ip-10-0-133-47.ec2.internal 48m Normal Scheduled pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7 to ip-10-0-133-47.ec2.internal 59m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 59m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 59m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-7765667bbf from 0 to 1 59m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 59m Normal SuccessfulCreate replicaset/router-default-7765667bbf Created pod: router-default-7765667bbf-4zpgd 52m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 52m Warning FailedMount pod/router-default-7765667bbf-4zpgd MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 52m Warning FailedMount pod/router-default-7765667bbf-4zpgd MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 52m Normal Pulled pod/router-default-7765667bbf-4zpgd Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/router-default-7765667bbf-4zpgd Add eth0 [10.132.0.10/23] from ovn-kubernetes 52m Normal Started pod/router-default-7765667bbf-4zpgd Container started 52m Normal Created pod/router-default-7765667bbf-4zpgd Container created 48m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 48m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 48m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-7p9tg 48m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-7p9tg Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 48m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-7p9tg Add eth0 [10.133.0.14/23] from ovn-kubernetes 48m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-7p9tg Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.478s (3.478s including waiting). Image size: 185476477 bytes. 48m Normal Created pod/istiod-openshift-gateway-7876bfcb74-7p9tg Container created 48m Normal Started pod/istiod-openshift-gateway-7876bfcb74-7p9tg Container started 48m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-7p9tg Readiness probe error: HTTP probe failed with statuscode: 503... 48m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-7p9tg Readiness probe failed: HTTP probe failed with statuscode: 503 48m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-77f556d647 from 0 to 1 48m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 48m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-b9qj7 48m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-689986b98c Created pod: openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn 48m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-689986b98c from 0 to 1 48m Normal Pulling pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 48m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Add eth0 [10.134.0.22/23] from ovn-kubernetes 48m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Add eth0 [10.134.0.23/23] from ovn-kubernetes 48m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 48m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 48m Normal Started pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Container started 48m Normal Created pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Container created 48m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 12.249s (12.249s including waiting). Image size: 183555333 bytes. 48m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 13.071s (13.071s including waiting). Image size: 183555333 bytes. 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedMount pod/kube-auth-proxy-69dd9bcffb-ddqsh MountVolume.SetUp failed for volume "tls-certs" : secret "kube-auth-proxy-tls" not found 47m Warning FailedMount pod/kube-auth-proxy-69dd9bcffb-zsxmb MountVolume.SetUp failed for volume "tls-certs" : secret "kube-auth-proxy-tls" not found 47m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-57dc67465 Created pod: data-science-gateway-data-science-gateway-class-57dc67465-xvbrd 47m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-57dc67465 from 0 to 1 47m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-69dd9bcffb from 0 to 2 47m Normal SuccessfulCreate replicaset/kube-auth-proxy-69dd9bcffb Created pod: kube-auth-proxy-69dd9bcffb-zsxmb 47m Normal SuccessfulCreate replicaset/kube-auth-proxy-69dd9bcffb Created pod: kube-auth-proxy-69dd9bcffb-ddqsh 47m Normal AddedInterface pod/kube-auth-proxy-69dd9bcffb-ddqsh Add eth0 [10.133.0.16/23] from ovn-kubernetes 47m Normal Started pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Container started 47m Normal Created pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Container created 47m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Add eth0 [10.134.0.32/23] from ovn-kubernetes 47m Normal Pulling pod/kube-auth-proxy-69dd9bcffb-ddqsh Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 47m Normal Pulled pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/kube-auth-proxy-69dd9bcffb-zsxmb Add eth0 [10.132.0.43/23] from ovn-kubernetes 47m Normal Pulling pod/kube-auth-proxy-69dd9bcffb-zsxmb Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 47m Normal Started pod/kube-auth-proxy-69dd9bcffb-ddqsh Container started 47m Normal Created pod/kube-auth-proxy-69dd9bcffb-ddqsh Container created 47m Normal Pulled pod/kube-auth-proxy-69dd9bcffb-ddqsh Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.223s (2.223s including waiting). Image size: 188904677 bytes. 47m Normal Pulled pod/kube-auth-proxy-69dd9bcffb-zsxmb Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.623s (2.623s including waiting). Image size: 188904677 bytes. 47m Normal Created pod/kube-auth-proxy-69dd9bcffb-zsxmb Container created 47m Normal Started pod/kube-auth-proxy-69dd9bcffb-zsxmb Container started 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Container created 46m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Container started 46m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 46m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-t7ntp 46m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Add eth0 [10.133.0.20/23] from ovn-kubernetes 46m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 46m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Readiness probe error: HTTP probe failed with statuscode: 503... 46m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Readiness probe failed: HTTP probe failed with statuscode: 503 46m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-7p9tg Stopping container discovery 46m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-7p9tg 46m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Container created 45m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-77f556d647 from 1 to 4 45m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Add eth0 [10.133.0.23/23] from ovn-kubernetes 45m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-hn66q Add eth0 [10.132.0.50/23] from ovn-kubernetes 45m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Add eth0 [10.134.0.44/23] from ovn-kubernetes 45m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 45m Normal Pulling pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 45m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-hn66q 45m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Pulling pod/maas-default-gateway-openshift-default-77f556d647-hn66q Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 45m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-cxtkt 45m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-vhrw7 45m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Container started 45m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Container started 45m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Container created 45m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-hn66q Container started 45m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.676s (2.676s including waiting). Image size: 183555333 bytes. 45m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-hn66q Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.89s (2.89s including waiting). Image size: 183555333 bytes. 45m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Container created 45m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Container started 45m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-hn66q Container created 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Startup probe error: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Startup probe failed: Get "http://10.133.0.23:15021/healthz/ready": dial tcp 10.133.0.23:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Startup probe error: Get "http://10.133.0.23:15021/healthz/ready": dial tcp 10.133.0.23:15021: connect: connection refused... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Startup probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-hn66q Startup probe error: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Startup probe failed: Get "http://10.134.0.22:15021/healthz/ready": dial tcp 10.134.0.22:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Startup probe error: Get "http://10.134.0.22:15021/healthz/ready": dial tcp 10.134.0.22:15021: connect: connection refused... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-hn66q Startup probe failed: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Add eth0 [10.134.0.45/23] from ovn-kubernetes 44m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Add eth0 [10.133.0.24/23] from ovn-kubernetes 44m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-77f556d647 from 4 to 7 44m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Container started 44m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-pxcpb 44m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-n789k 44m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-bb5z2 44m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-n789k Container started 44m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-n789k Add eth0 [10.132.0.52/23] from ovn-kubernetes 44m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: 44m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Container created 44m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-n789k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-n789k Container created 44m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Container started 44m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Container created 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Startup probe failed: Get "http://10.133.0.24:15021/healthz/ready": dial tcp 10.133.0.24:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-pxcpb Startup probe error: Get "http://10.133.0.24:15021/healthz/ready": dial tcp 10.133.0.24:15021: connect: connection refused... 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-n789k Startup probe error: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-n789k Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Startup probe error: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused... 43m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-77f556d647 from 7 to 10 43m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-4bknm 43m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 (combined from similar events): Created pod: maas-default-gateway-openshift-default-77f556d647-vftgt 43m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-wmxgt 43m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-4bknm Add eth0 [10.134.0.52/23] from ovn-kubernetes 43m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Container started 43m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Container created 43m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Add eth0 [10.133.0.27/23] from ovn-kubernetes 43m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-vftgt Add eth0 [10.132.0.53/23] from ovn-kubernetes 43m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-vftgt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-vftgt Container started 43m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-vftgt Container created 43m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-4bknm Container started 43m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-4bknm Container created 43m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-4bknm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-vftgt Startup probe error: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-vftgt Startup probe failed: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused 43m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Startup probe error: Get "http://10.133.0.27:15021/healthz/ready": dial tcp 10.133.0.27:15021: connect: connection refused... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Startup probe failed: Get "http://10.133.0.27:15021/healthz/ready": dial tcp 10.133.0.27:15021: connect: connection refused 43m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-4bknm Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused... 43m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-4bknm Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused 43m Warning FailedCreate replicaset/payload-processing-5f78687d5b Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 43m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5f78687d5b from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 43m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 43m Normal Pulling pod/payload-pre-processing-7b98bb678c-9w8hc Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 43m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-9w8hc 43m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-9w8hc Add eth0 [10.133.0.29/23] from ovn-kubernetes 43m Normal AddedInterface pod/payload-processing-5f78687d5b-j6n87 Add eth0 [10.134.0.54/23] from ovn-kubernetes 43m Normal SuccessfulCreate replicaset/payload-processing-5f78687d5b Created pod: payload-processing-5f78687d5b-j6n87 43m Normal Pulling pod/payload-processing-5f78687d5b-j6n87 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 43m Normal Pulled pod/payload-pre-processing-7b98bb678c-9w8hc Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.234s (2.234s including waiting). Image size: 125317101 bytes. 43m Normal Started pod/payload-pre-processing-7b98bb678c-9w8hc Container started 43m Normal Created pod/payload-pre-processing-7b98bb678c-9w8hc Container created 43m Normal Pulled pod/payload-processing-5f78687d5b-j6n87 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.561s (3.561s including waiting). Image size: 125317101 bytes. 43m Normal Created pod/payload-processing-5f78687d5b-j6n87 Container created 43m Normal Started pod/payload-processing-5f78687d5b-j6n87 Container started 42m Normal AddedLabel gateway/e2e-worker-w4-4ef6ac Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-4ef6ac 42m Normal AddedLabel gateway/e2e-worker-w5-fdf4da Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-fdf4da 42m Normal AddedLabel gateway/e2e-worker-w0-6ce5eb Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-6ce5eb 42m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled up replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw 42m Normal AddedLabel gateway/e2e-worker-w2-fde2c5 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-fde2c5 42m Normal AddedLabel gateway/e2e-worker-w6-ef13cc Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-ef13cc 42m Normal AddedLabel gateway/e2e-worker-w3-8b6cf1 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-8b6cf1 42m Normal AddedLabel gateway/e2e-worker-w1-fb2fa0 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-fb2fa0 42m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled up replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 0 to 1 42m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx 42m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w 42m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Add eth0 [10.133.0.30/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx 42m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 42m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 0 to 1 42m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled up replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 42m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Add eth0 [10.134.0.56/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 0 to 1 42m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled up replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 0 to 1 42m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Add eth0 [10.133.0.31/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 42m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Add eth0 [10.133.0.32/23] from ovn-kubernetes 42m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Add eth0 [10.134.0.58/23] from ovn-kubernetes 42m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Add eth0 [10.134.0.59/23] from ovn-kubernetes 42m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Add eth0 [10.134.0.57/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-fde2c5 Scaled up replica set payload-processing-e2e-worker-w2-fde2c5-76f6995c88 from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-fde2c5 Scaled up replica set payload-pre-processing-e2e-worker-w2-fde2c5-7546567987 from 0 to 1 42m Normal Started pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Container started 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987 Error creating: pods "payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-fde2c5: serviceaccount "payload-processing-e2e-worker-w2-fde2c5" not found 42m Normal Pulled pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Container created 42m Normal AddedInterface pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Add eth0 [10.133.0.34/23] from ovn-kubernetes 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-fde2c5-76f6995c88 Error creating: pods "payload-processing-e2e-worker-w2-fde2c5-76f6995c88-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-fde2c5: serviceaccount "payload-processing-e2e-worker-w2-fde2c5" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987 Created pod: payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl 42m Normal Started pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Container started 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Add eth0 [10.133.0.33/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Container created 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-fde2c5-76f6995c88 Created pod: payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-fb2fa0 Scaled up replica set payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff Error creating: pods "payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0: serviceaccount "payload-processing-e2e-worker-w1-fb2fa0" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-fb2fa0 Scaled up replica set payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4 from 0 to 1 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4 Error creating: pods "payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0: serviceaccount "payload-processing-e2e-worker-w1-fb2fa0" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff Created pod: payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj 42m Normal AddedInterface pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Add eth0 [10.133.0.36/23] from ovn-kubernetes 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Add eth0 [10.133.0.35/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4 Created pod: payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d 42m Normal Created pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Container created 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-fdf4da Scaled up replica set payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c from 0 to 1 42m Normal Started pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Container started 42m Normal Started pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Container started 42m Normal Created pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Container created 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-fdf4da Scaled up replica set payload-processing-e2e-worker-w5-fdf4da-55966b9c98 from 0 to 1 42m Normal Pulled pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Pulled pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648 Error creating: pods "payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1: serviceaccount "payload-processing-e2e-worker-w3-8b6cf1" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c Created pod: payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c Error creating: pods "payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-fdf4da: serviceaccount "payload-processing-e2e-worker-w5-fdf4da" not found 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-fdf4da-55966b9c98 Error creating: pods "payload-processing-e2e-worker-w5-fdf4da-55966b9c98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-fdf4da: serviceaccount "payload-processing-e2e-worker-w5-fdf4da" not found 42m Normal Started pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Container started 42m Normal Created pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Container created 42m Normal Started pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Container started 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67 Error creating: pods "payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1: serviceaccount "payload-processing-e2e-worker-w3-8b6cf1" not found 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-8b6cf1 Scaled up replica set payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-fdf4da-55966b9c98 Created pod: payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b 42m Normal Created pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Container created 42m Normal AddedInterface pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Add eth0 [10.133.0.38/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Add eth0 [10.133.0.37/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-8b6cf1 Scaled up replica set payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648 from 0 to 1 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Add eth0 [10.134.0.61/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67 Created pod: payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt 42m Normal Started pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Container started 42m Normal Created pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Container created 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Container created 42m Normal Started pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Container started 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648 Created pod: payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 42m Normal AddedInterface pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Add eth0 [10.133.0.39/23] from ovn-kubernetes 42m Normal Pulled pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574 Error creating: pods "payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb: serviceaccount "payload-processing-e2e-worker-w0-6ce5eb" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574 Created pod: payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-6ce5eb Scaled up replica set payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff Error creating: pods "payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb: serviceaccount "payload-processing-e2e-worker-w0-6ce5eb" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff Created pod: payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-6ce5eb Scaled up replica set payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574 from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-ef13cc Scaled up replica set payload-processing-e2e-worker-w6-ef13cc-755485dc8d from 0 to 1 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Add eth0 [10.134.0.63/23] from ovn-kubernetes 42m Normal Created pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Container created 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-ef13cc-755485dc8d Created pod: payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Add eth0 [10.134.0.62/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-ef13cc Scaled up replica set payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595 from 0 to 1 42m Normal Started pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Container started 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595 Created pod: payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595 Error creating: pods "payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-ef13cc: serviceaccount "payload-processing-e2e-worker-w6-ef13cc" not found 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-ef13cc-755485dc8d Error creating: pods "payload-processing-e2e-worker-w6-ef13cc-755485dc8d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-ef13cc: serviceaccount "payload-processing-e2e-worker-w6-ef13cc" not found 42m Normal Pulled pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Started pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Container started 42m Normal Created pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Container created 42m Normal Created pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Container created 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Add eth0 [10.133.0.41/23] from ovn-kubernetes 42m Normal Started pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Container started 42m Normal Started pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Container started 42m Normal Pulled pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Container created 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Add eth0 [10.133.0.40/23] from ovn-kubernetes 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87 Error creating: pods "payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac: serviceaccount "payload-processing-e2e-worker-w4-4ef6ac" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87 Created pod: payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-4ef6ac Scaled up replica set payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87 from 0 to 1 42m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976 Error creating: pods "payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac: serviceaccount "payload-processing-e2e-worker-w4-4ef6ac" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976 Created pod: payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-4ef6ac Scaled up replica set payload-processing-e2e-worker-w4-4ef6ac-cbfff8976 from 0 to 1 42m Normal Started pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Container started 42m Normal AddedInterface pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Add eth0 [10.133.0.42/23] from ovn-kubernetes 42m Normal Pulled pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Add eth0 [10.134.0.64/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Container created 42m Normal Started pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Container started 42m Normal Created pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Container created 42m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Container created 42m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Container started 42m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Startup probe error: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Startup probe failed: Get "http://10.133.0.30:15021/healthz/ready": dial tcp 10.133.0.30:15021: connect: connection refused 42m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Container created 42m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Container created 42m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Container started 42m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Container started 41m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Container created 41m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Container started 41m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Container created 41m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Container started 41m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Container started 41m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Startup probe failed: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Startup probe error: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused... 41m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused... 41m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Container created 41m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled up replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 1 to 4 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Container created 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Add eth0 [10.133.0.45/23] from ovn-kubernetes 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Container created 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Container started 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Add eth0 [10.132.0.59/23] from ovn-kubernetes 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Container started 41m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused... 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Add eth0 [10.134.0.66/23] from ovn-kubernetes 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Container created 41m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Startup probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Startup probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 41m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Container started 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Startup probe failed: Get "http://10.132.0.59:15021/healthz/ready": dial tcp 10.132.0.59:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused... 41m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused 41m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Startup probe error: Get "http://10.132.0.59:15021/healthz/ready": dial tcp 10.132.0.59:15021: connect: connection refused... 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Container created 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Container started 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Startup probe error: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Startup probe failed: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Add eth0 [10.133.0.49/23] from ovn-kubernetes 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 41m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled up replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 1 to 2 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 41m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Container created 41m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm 41m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd 41m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled up replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 4 to 7 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 7; reason: 41m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Container started 41m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 1 to 2 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Container created 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Container created 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Container started 41m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Add eth0 [10.133.0.50/23] from ovn-kubernetes 41m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Container started 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Container created 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z 41m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Container started 41m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Add eth0 [10.134.0.69/23] from ovn-kubernetes 41m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Container created 41m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Container started 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Add eth0 [10.133.0.51/23] from ovn-kubernetes 41m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr 41m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Add eth0 [10.132.0.64/23] from ovn-kubernetes 41m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Startup probe error: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused... 41m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Startup probe failed: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused 41m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Startup probe failed: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Startup probe error: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused... 41m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Startup probe error: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Startup probe failed: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused 40m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst 40m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 1 to 2 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Container started 40m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Container created 40m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Add eth0 [10.133.0.52/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 40m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d 40m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 2 to 3 40m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Container started 40m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Container created 40m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Add eth0 [10.132.0.66/23] from ovn-kubernetes 40m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Startup probe error: Get "http://10.132.0.66:15021/healthz/ready": dial tcp 10.132.0.66:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Startup probe failed: Get "http://10.132.0.66:15021/healthz/ready": dial tcp 10.132.0.66:15021: connect: connection refused 40m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled up replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 2 to 4 40m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled up replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 1 to 2 40m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 2 to 3 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 40m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Add eth0 [10.132.0.67/23] from ovn-kubernetes 40m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q 40m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv 40m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 40m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Add eth0 [10.134.0.73/23] from ovn-kubernetes 40m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Container created 40m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Add eth0 [10.134.0.72/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Add eth0 [10.132.0.68/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Container created 40m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Container started 40m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Container started 40m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Container started 40m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Container created 40m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Container created 40m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Container started 40m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 40m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 40m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 40m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused 40m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 40m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled up replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 1 to 2 39m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 39m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q 39m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf 39m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Container created 39m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Container started 39m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled up replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 7 to 10 39m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Add eth0 [10.134.0.74/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 (combined from similar events): Created pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 39m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Add eth0 [10.132.0.69/23] from ovn-kubernetes 39m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Add eth0 [10.133.0.58/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Container created 39m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Container started 39m Normal AddedInterface pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Add eth0 [10.134.0.75/23] from ovn-kubernetes 39m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Container started 39m Normal Pulled pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Container created 39m Normal Started pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Container started 39m Normal Created pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Container created 39m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Startup probe failed: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Startup probe error: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused... 39m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Readiness probe error: Get "http://10.134.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Readiness probe failed: Get "http://10.134.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 3 to 4 39m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 39m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g 39m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 39m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 3 to 6 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 39m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k 39m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Add eth0 [10.133.0.66/23] from ovn-kubernetes 39m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Container created 39m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Container started 39m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Container started 39m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Container started 39m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Container created 39m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Container created 39m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Add eth0 [10.132.0.70/23] from ovn-kubernetes 39m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Readiness probe error: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Readiness probe failed: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Add eth0 [10.133.0.67/23] from ovn-kubernetes 39m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Readiness probe failed: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Readiness probe error: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Readiness probe failed: timeout: failed to connect service "10.134.0.63:50052" within 1s: context deadline exceeded 39m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Add eth0 [10.134.0.77/23] from ovn-kubernetes 39m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Container started 39m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Container created 39m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 39m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Readiness probe failed: Get "http://10.134.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w 38m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r 38m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled up replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 4 to 8 38m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 38m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc 38m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Container started 38m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Add eth0 [10.132.0.72/23] from ovn-kubernetes 38m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Add eth0 [10.133.0.76/23] from ovn-kubernetes 38m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Container created 38m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Add eth0 [10.133.0.77/23] from ovn-kubernetes 38m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Add eth0 [10.134.0.79/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Container started 38m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Container created 38m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Container started 38m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Container created 38m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Container created 38m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Container started 38m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Startup probe error: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused... 38m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Startup probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Startup probe failed: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Startup probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 38m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 38m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 38m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf 38m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d 38m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh 38m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 1 to 2 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 38m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled up replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 2 to 4 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 38m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 38m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 4 to 8 38m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 38m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Container created 38m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Container started 38m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Container created 38m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Add eth0 [10.133.0.80/23] from ovn-kubernetes 38m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Add eth0 [10.134.0.82/23] from ovn-kubernetes 38m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Container created 38m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Add eth0 [10.132.0.74/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Container started 38m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Add eth0 [10.134.0.81/23] from ovn-kubernetes 38m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Add eth0 [10.134.0.80/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Container started 38m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Container started 38m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Add eth0 [10.134.0.83/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Container started 38m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Container created 38m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Add eth0 [10.132.0.75/23] from ovn-kubernetes 38m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Container created 38m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Container created 38m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Container started 38m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Container created 38m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Container started 38m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Startup probe error: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused... 38m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Startup probe failed: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Startup probe failed: Get "http://10.134.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Startup probe error: Get "http://10.134.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Readiness probe failed: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Readiness probe error: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Startup probe failed: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Startup probe failed: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Startup probe error: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Startup probe failed: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Startup probe failed: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Startup probe error: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused... 38m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 38m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Startup probe error: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused... 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 5; reason: 38m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled up replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 4 to 5 38m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Startup probe error: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused... 38m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Add eth0 [10.133.0.82/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Container started 38m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Container created 38m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 38m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 38m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 38m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled up replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 2 to 4 38m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Container created 38m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Add eth0 [10.132.0.78/23] from ovn-kubernetes 38m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Add eth0 [10.134.0.84/23] from ovn-kubernetes 38m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Container started 38m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Container created 38m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Container started 38m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Readiness probe error: Get "http://10.133.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Readiness probe failed: Get "http://10.133.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe error: Get "http://10.134.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Startup probe failed: Get "http://10.134.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe failed: Get "http://10.134.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Startup probe error: Get "http://10.134.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Startup probe failed: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default New size: 5; reason: 37m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled up replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 4 to 5 37m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 37m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 6 to 8 37m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Startup probe error: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused... 37m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 37m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 37m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Container started 37m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Add eth0 [10.134.0.88/23] from ovn-kubernetes 37m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Container created 37m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Add eth0 [10.133.0.88/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Container created 37m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Container started 37m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Add eth0 [10.133.0.89/23] from ovn-kubernetes 37m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Container started 37m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Container created 37m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Startup probe failed: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused 37m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Startup probe failed: Get "http://10.133.0.88:15021/healthz/ready": dial tcp 10.133.0.88:15021: connect: connection refused 37m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Startup probe error: Get "http://10.133.0.88:15021/healthz/ready": dial tcp 10.133.0.88:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Startup probe error: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Startup probe failed: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused 37m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Startup probe error: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused... 37m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 (combined from similar events): Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 37m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 2 to 4 37m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 37m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 37m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled up replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 8 to 10 37m Normal SuccessfulCreate replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Created pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf 37m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Add eth0 [10.132.0.80/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Add eth0 [10.133.0.91/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Add eth0 [10.132.0.81/23] from ovn-kubernetes 37m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Container started 37m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Container created 37m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Add eth0 [10.134.0.90/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Container created 37m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Container started 37m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Container created 37m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Container started 37m Normal Started pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Container started 37m Normal Created pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Container created 37m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 37m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused 37m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 37m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Startup probe failed: Get "http://10.134.0.90:15021/healthz/ready": dial tcp 10.134.0.90:15021: connect: connection refused 37m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Startup probe error: Get "http://10.134.0.90:15021/healthz/ready": dial tcp 10.134.0.90:15021: connect: connection refused... 37m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 (combined from similar events): Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 37m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled up replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 8 to 10 37m Normal SuccessfulCreate replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Created pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd 37m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Add eth0 [10.132.0.82/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Add eth0 [10.134.0.92/23] from ovn-kubernetes 37m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Container started 37m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Container created 37m Normal Pulled pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Container created 37m Normal Started pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Container started 37m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Readiness probe error: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Readiness probe failed: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Readiness probe error: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 37m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Readiness probe failed: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Startup probe failed: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused 37m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 37m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-4drwq 37m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 37m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 37m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Startup probe error: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused... 37m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Add eth0 [10.134.0.93/23] from ovn-kubernetes 37m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.968s (3.968s including waiting). Image size: 185476477 bytes. 37m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Container created 37m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Container started 37m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled up replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 5 to 10 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 36m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 (combined from similar events): Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj 36m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 (combined from similar events): Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 36m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp 36m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr 36m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt 36m Normal SuccessfulCreate replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Created pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn 36m Normal SuccessfulCreate replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Created pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 36m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled up replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 8 to 10 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 36m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Add eth0 [10.132.0.84/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Container created 36m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Container started 36m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Add eth0 [10.132.0.85/23] from ovn-kubernetes 36m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Add eth0 [10.132.0.83/23] from ovn-kubernetes 36m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Container created 36m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Container started 36m Normal Created pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Container created 36m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Container started 36m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Add eth0 [10.133.0.96/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Container created 36m Normal Started pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Container started 36m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Container created 36m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Container started 36m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Container created 36m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Container started 36m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Add eth0 [10.133.0.98/23] from ovn-kubernetes 36m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Add eth0 [10.133.0.97/23] from ovn-kubernetes 36m Normal Started pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Container started 36m Normal AddedInterface pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Add eth0 [10.134.0.94/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Container created 36m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Startup probe error: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Startup probe failed: Get "http://10.133.0.97:15021/healthz/ready": dial tcp 10.133.0.97:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Startup probe failed: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Startup probe error: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Startup probe error: Get "http://10.133.0.97:15021/healthz/ready": dial tcp 10.133.0.97:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused... 36m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 36m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Startup probe failed: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Startup probe error: Get "http://10.134.0.94:15021/healthz/ready": dial tcp 10.134.0.94:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Startup probe failed: Get "http://10.134.0.94:15021/healthz/ready": dial tcp 10.134.0.94:15021: connect: connection refused 36m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled up replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 5 to 10 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 36m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 36m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 (combined from similar events): Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 36m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 36m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz 36m Normal SuccessfulCreate replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Created pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p 36m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Readiness probe error: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Readiness probe failed: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Add eth0 [10.134.0.95/23] from ovn-kubernetes 36m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Container started 36m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Add eth0 [10.132.0.86/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Container created 36m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Container started 36m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Container created 36m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Add eth0 [10.132.0.87/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Container started 36m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Container created 36m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Add eth0 [10.133.0.104/23] from ovn-kubernetes 36m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Container created 36m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Add eth0 [10.133.0.105/23] from ovn-kubernetes 36m Normal Created pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Container created 36m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Container started 36m Normal Started pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Container started 36m Normal Pulled pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 36m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 36m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Startup probe failed: Get "http://10.134.0.95:15021/healthz/ready": dial tcp 10.134.0.95:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Startup probe error: Get "http://10.134.0.95:15021/healthz/ready": dial tcp 10.134.0.95:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Startup probe failed: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Startup probe error: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused... 36m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Startup probe error: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Startup probe failed: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused 36m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Readiness probe error: Get "http://10.134.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Readiness probe error: Get "http://10.132.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Readiness probe failed: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Readiness probe error: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Readiness probe failed: Get "http://10.132.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Readiness probe failed: Get "http://10.133.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Readiness probe failed: Get "http://10.133.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Readiness probe error: Get "http://10.133.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Readiness probe error: Get "http://10.133.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Readiness probe failed: Get "http://10.133.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Readiness probe error: Get "http://10.133.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Container started 35m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Container created 35m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Container created 35m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Container started 35m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Add eth0 [10.132.0.88/23] from ovn-kubernetes 35m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Container created 35m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 4 to 7 35m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Add eth0 [10.134.0.97/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 35m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Add eth0 [10.133.0.116/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 35m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 35m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Container started 35m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Startup probe failed: Get "http://10.134.0.97:15021/healthz/ready": dial tcp 10.134.0.97:15021: connect: connection refused 35m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 35m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Startup probe error: Get "http://10.133.0.116:15021/healthz/ready": dial tcp 10.133.0.116:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Startup probe failed: Get "http://10.133.0.116:15021/healthz/ready": dial tcp 10.133.0.116:15021: connect: connection refused 35m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 35m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Startup probe error: Get "http://10.134.0.97:15021/healthz/ready": dial tcp 10.134.0.97:15021: connect: connection refused... 35m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 9; reason: cpu resource utilization (percentage of request) above target 35m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 7 to 9 35m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw 35m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Add eth0 [10.132.0.89/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Container created 35m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Container started 35m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Add eth0 [10.133.0.119/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Container created 35m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Container started 35m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused 35m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Startup probe failed: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused 35m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Startup probe error: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused... 35m Normal SuccessfulCreate replicaset/e2e-migrate-535b743b-gw-openshift-default-f97c88957 Created pod: e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w 35m Normal AddedLabel gateway/e2e-migrate-535b743b-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-535b743b-gw 35m Normal ScalingReplicaSet deployment/e2e-migrate-535b743b-gw-openshift-default Scaled up replica set e2e-migrate-535b743b-gw-openshift-default-f97c88957 from 0 to 1 35m Normal AddedInterface pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Add eth0 [10.133.0.126/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-535b743b Scaled up replica set payload-processing-e2e-migrate-535b743b-c687fddc4 from 0 to 1 35m Warning FailedCreate replicaset/payload-processing-e2e-migrate-535b743b-c687fddc4 Error creating: pods "payload-processing-e2e-migrate-535b743b-c687fddc4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-535b743b: serviceaccount "payload-processing-e2e-migrate-535b743b" not found 35m Normal SuccessfulCreate replicaset/e2e-ait-7cb9c50e-openshift-default-6995c464bf Created pod: e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p 35m Normal ScalingReplicaSet deployment/e2e-ait-7cb9c50e-openshift-default Scaled up replica set e2e-ait-7cb9c50e-openshift-default-6995c464bf from 0 to 1 35m Normal AddedLabel gateway/e2e-ait-7cb9c50e Added label istio.io/rev=openshift-gateway to gateway e2e-ait-7cb9c50e 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c Error creating: pods "payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-535b743b: serviceaccount "payload-processing-e2e-migrate-535b743b" not found 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c Created pod: payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-535b743b Scaled up replica set payload-pre-processing-e2e-migrate-535b743b-657bf56d4c from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-535b743b-c687fddc4 Created pod: payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr 35m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Add eth0 [10.134.0.99/23] from ovn-kubernetes 35m Normal AddedInterface pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Add eth0 [10.134.0.100/23] from ovn-kubernetes 35m Normal Created pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Container created 35m Normal Pulled pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Add eth0 [10.134.0.101/23] from ovn-kubernetes 35m Normal Started pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Container started 35m Normal Started pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Container started 35m Normal Pulled pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Created pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Container created 35m Normal AddedLabel gateway/e2e-ait-f6a224e7 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-f6a224e7 35m Normal SuccessfulCreate replicaset/e2e-ait-f6a224e7-openshift-default-7d8f55f89f Created pod: e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf 35m Normal ScalingReplicaSet deployment/e2e-ait-f6a224e7-openshift-default Scaled up replica set e2e-ait-f6a224e7-openshift-default-7d8f55f89f from 0 to 1 35m Normal Created pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Container created 35m Normal Pulled pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Add eth0 [10.134.0.102/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579 Created pod: payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-7cb9c50e Scaled up replica set payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579 from 0 to 1 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579 Error creating: pods "payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-7cb9c50e: serviceaccount "payload-processing-e2e-ait-7cb9c50e" not found 35m Warning FailedCreate replicaset/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6 Error creating: pods "payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-7cb9c50e: serviceaccount "payload-processing-e2e-ait-7cb9c50e" not found 35m Normal Started pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Container started 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-7cb9c50e Scaled up replica set payload-processing-e2e-ait-7cb9c50e-68dd7c7db6 from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6 Created pod: payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx 34m Normal AddedInterface pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Add eth0 [10.132.0.92/23] from ovn-kubernetes 34m Normal Created pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Container created 34m Normal Pulling pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 34m Normal Pulled pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal AddedInterface pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Add eth0 [10.133.0.128/23] from ovn-kubernetes 34m Normal Started pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Container started 34m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Readiness probe failed: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Readiness probe error: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Readiness probe error: Get "http://10.133.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Readiness probe failed: Get "http://10.134.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Readiness probe failed: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Readiness probe error: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Readiness probe error: Get "http://10.134.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Readiness probe failed: Get "http://10.133.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Readiness probe failed: Get "http://10.133.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Readiness probe error: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Readiness probe failed: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Readiness probe error: Get "http://10.133.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-7cb9c50e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-7cb9c50e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Readiness probe failed: timeout: health rpc did not complete within 1s 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Readiness probe failed: Get "http://10.133.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-f6a224e7 Scaled up replica set payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5 from 0 to 1 34m Warning FailedCreate replicaset/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4 Error creating: pods "payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-f6a224e7: serviceaccount "payload-processing-e2e-ait-f6a224e7" not found 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Readiness probe error: Get "http://10.133.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4 Created pod: payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-f6a224e7 Scaled up replica set payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4 from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5 Created pod: payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5 Error creating: pods "payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-f6a224e7: serviceaccount "payload-processing-e2e-ait-f6a224e7" not found 34m Normal Created pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Container created 34m Normal Created pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Container created 34m Normal AddedInterface pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Add eth0 [10.132.0.93/23] from ovn-kubernetes 34m Normal AddedInterface pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Add eth0 [10.132.0.94/23] from ovn-kubernetes 34m Normal Pulled pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Container started 34m Normal Pulled pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 9.122s (9.122s including waiting). Image size: 125317101 bytes. 34m Normal Started pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Container started 34m Normal Pulling pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 34m Normal Pulled pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 407ms (407ms including waiting). Image size: 125317101 bytes. 34m Normal Started pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Container started 34m Normal Created pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Container created 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-535b743b-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-535b743b-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 34m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled up replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 9 to 10 34m Normal SuccessfulCreate replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 (combined from similar events): Created pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk 34m Normal Started pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Container started 34m Normal AddedInterface pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Add eth0 [10.133.0.133/23] from ovn-kubernetes 34m Normal Pulled pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Container created 34m Normal AddedLabel gateway/e2e-derive-a90958fa Added label istio.io/rev=openshift-gateway to gateway e2e-derive-a90958fa 34m Normal ScalingReplicaSet deployment/e2e-derive-a90958fa-openshift-default Scaled up replica set e2e-derive-a90958fa-openshift-default-844f8b6d7c from 0 to 1 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-f6a224e7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-f6a224e7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal SuccessfulCreate replicaset/e2e-derive-a90958fa-openshift-default-844f8b6d7c Created pod: e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Startup probe error: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Startup probe failed: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused 34m Normal AddedInterface pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Add eth0 [10.133.0.134/23] from ovn-kubernetes 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-a90958fa Scaled up replica set payload-pre-processing-e2e-derive-a90958fa-649c4cb67f from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-a90958fa Scaled up replica set payload-processing-e2e-derive-a90958fa-755c99bc46 from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f Created pod: payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f Error creating: pods "payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-a90958fa: serviceaccount "payload-processing-e2e-derive-a90958fa" not found 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-a90958fa-755c99bc46 Created pod: payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj 34m Warning FailedCreate replicaset/payload-processing-e2e-derive-a90958fa-755c99bc46 Error creating: pods "payload-processing-e2e-derive-a90958fa-755c99bc46-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-a90958fa: serviceaccount "payload-processing-e2e-derive-a90958fa" not found 34m Normal Pulled pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal AddedInterface pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Add eth0 [10.133.0.135/23] from ovn-kubernetes 34m Normal Started pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Container started 34m Normal Created pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Container created 34m Normal Started pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Container started 34m Normal AddedInterface pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Add eth0 [10.133.0.136/23] from ovn-kubernetes 34m Normal Pulled pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Container created 34m Normal AddedLabel gateway/e2e-shared-a-w3-b2b596 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-b2b596 34m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-b2b596-openshift-default Scaled up replica set e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 from 0 to 1 34m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr 34m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Add eth0 [10.134.0.104/23] from ovn-kubernetes 34m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Readiness probe error: Get "http://10.133.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Readiness probe failed: Get "http://10.133.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Readiness probe failed: Get "http://10.134.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Readiness probe error: Get "http://10.134.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/payload-processing-5f78687d5b-j6n87 Readiness probe failed: timeout: failed to connect service "10.134.0.54:50052" within 1s: context deadline exceeded 34m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Readiness probe error: Get "http://10.134.0.90:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Readiness probe failed: Get "http://10.134.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Readiness probe error: Get "http://10.134.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Readiness probe failed: Get "http://10.134.0.90:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Readiness probe failed: timeout: failed to connect service "10.134.0.100:50052" within 1s: context deadline exceeded 34m Warning Unhealthy pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Readiness probe failed: Get "http://10.134.0.95:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Readiness probe error: Get "http://10.134.0.95:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 34m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 34m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-xttzf 34m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 34m Warning Unhealthy pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Readiness probe failed: Get "http://10.133.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Add eth0 [10.132.0.97/23] from ovn-kubernetes 34m Warning ProbeError pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Readiness probe error: Get "http://10.133.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal SuccessfulCreate replicaset/e2e-ait-7cb9c50e-openshift-default-6995c464bf Created pod: e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 34m Normal SuccessfulCreate replicaset/e2e-ait-7cb9c50e-openshift-default-6995c464bf Created pod: e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ait-7cb9c50e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 34m Normal AddedInterface pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Add eth0 [10.132.0.98/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/e2e-ait-7cb9c50e-openshift-default-6995c464bf Created pod: e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf 34m Normal Pulled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal AddedInterface pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Add eth0 [10.133.0.139/23] from ovn-kubernetes 34m Normal ScalingReplicaSet deployment/e2e-ait-7cb9c50e-openshift-default Scaled up replica set e2e-ait-7cb9c50e-openshift-default-6995c464bf from 1 to 4 34m Normal Started pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Container started 34m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Container started 34m Normal Pulled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Pulled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Container created 34m Normal Started pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Container started 34m Normal AddedInterface pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Add eth0 [10.133.0.138/23] from ovn-kubernetes 34m Normal Created pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Container created 34m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.692s (2.692s including waiting). Image size: 185476477 bytes. 34m Normal Created pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Container created 34m Normal Started pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Container started 34m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Container created 34m Normal Started pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Container started 34m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Readiness probe error: HTTP probe failed with statuscode: 503... 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc Error creating: pods "payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596: serviceaccount "payload-processing-e2e-shared-a-w3-b2b596" not found 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58 Error creating: pods "payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596: serviceaccount "payload-processing-e2e-shared-a-w3-b2b596" not found 34m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Readiness probe failed: HTTP probe failed with statuscode: 503 34m Normal Pulled pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Container created 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc Created pod: payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-b2b596 Scaled up replica set payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58 Created pod: payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-b2b596 Scaled up replica set payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58 from 0 to 1 34m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Container created 34m Normal Started pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Container started 34m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Add eth0 [10.134.0.106/23] from ovn-kubernetes 34m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Container started 34m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Pulled pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Add eth0 [10.134.0.105/23] from ovn-kubernetes 34m Normal Created pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Container created 34m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Readiness probe failed: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Readiness probe error: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal Killing pod/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr Stopping container payload-processing 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-a90958fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-a90958fa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal Killing pod/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w Stopping container payload-pre-processing 34m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Startup probe error: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Startup probe failed: Get "http://10.132.0.98:15021/healthz/ready": dial tcp 10.132.0.98:15021: connect: connection refused 33m Normal Pulled pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Container created 33m Normal Started pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Container started 33m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Startup probe failed: Get "http://10.133.0.139:15021/healthz/ready": dial tcp 10.133.0.139:15021: connect: connection refused 33m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Startup probe error: Get "http://10.133.0.139:15021/healthz/ready": dial tcp 10.133.0.139:15021: connect: connection refused... 33m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Startup probe error: Get "http://10.133.0.138:15021/healthz/ready": dial tcp 10.133.0.138:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Startup probe failed: Get "http://10.133.0.138:15021/healthz/ready": dial tcp 10.133.0.138:15021: connect: connection refused 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-535b743b-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-535b743b-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning ProbeError pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Startup probe error: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Startup probe failed: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused 33m Normal Killing pod/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k Stopping container payload-pre-processing 33m Normal Killing pod/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx Stopping container payload-processing 33m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Startup probe error: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Startup probe failed: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-7cb9c50e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal Killing pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w Stopping container istio-proxy 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-7cb9c50e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal SuccessfulCreate replicaset/e2e-ait-b0171017-openshift-default-56f99f8d7 Created pod: e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr 33m Normal ScalingReplicaSet deployment/e2e-ait-b0171017-openshift-default Scaled up replica set e2e-ait-b0171017-openshift-default-56f99f8d7 from 0 to 1 33m Normal AddedLabel gateway/e2e-ait-b0171017 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-b0171017 33m Normal AddedInterface pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Add eth0 [10.134.0.108/23] from ovn-kubernetes 33m Normal Killing pod/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z Stopping container payload-processing 33m Normal Created pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Container created 33m Normal Pulled pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Killing pod/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt Stopping container payload-pre-processing 33m Normal Started pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Container started 33m Warning FailedCreate replicaset/payload-processing-e2e-ait-b0171017-8fc874664 Error creating: pods "payload-processing-e2e-ait-b0171017-8fc874664-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b0171017: serviceaccount "payload-processing-e2e-ait-b0171017" not found 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-b0171017 Scaled up replica set payload-pre-processing-e2e-ait-b0171017-695499df47 from 0 to 1 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-b0171017-695499df47 Error creating: pods "payload-pre-processing-e2e-ait-b0171017-695499df47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b0171017: serviceaccount "payload-processing-e2e-ait-b0171017" not found 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-b0171017-695499df47 Created pod: payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-b0171017-8fc874664 Created pod: payload-processing-e2e-ait-b0171017-8fc874664-qpttr 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-b0171017 Scaled up replica set payload-processing-e2e-ait-b0171017-8fc874664 from 0 to 1 33m Normal Pulled pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Killing pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 Stopping container istio-proxy 33m Normal Created pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Container created 33m Normal Pulled pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Killing pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p Stopping container istio-proxy 33m Normal Started pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Container started 33m Normal AddedInterface pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Add eth0 [10.133.0.143/23] from ovn-kubernetes 33m Normal Killing pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Stopping container istio-proxy 33m Normal AddedInterface pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Add eth0 [10.133.0.142/23] from ovn-kubernetes 33m Normal Created pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Container created 33m Normal Killing pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Stopping container istio-proxy 33m Normal Started pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Container started 33m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning Unhealthy pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal Killing pod/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf Stopping container istio-proxy 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-b0171017-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-b0171017-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning Unhealthy pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Readiness probe failed: Get "http://10.133.0.134:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 33m Warning ProbeError pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Readiness probe error: Get "http://10.133.0.134:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 33m Normal Started pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Container started 33m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 4 33m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 4; reason: cpu resource utilization (percentage of request) above target 33m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-2s5j4 33m Normal Pulled pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Container created 33m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Container started 33m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Add eth0 [10.133.0.145/23] from ovn-kubernetes 33m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Container created 33m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 33m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-b0171017-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-b0171017-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning Unhealthy pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Startup probe failed: Get "http://10.133.0.134:15021/healthz/ready": dial tcp 10.133.0.134:15021: connect: connection refused 33m Warning ProbeError pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Startup probe error: Get "http://10.133.0.134:15021/healthz/ready": dial tcp 10.133.0.134:15021: connect: connection refused... 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-a90958fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Killing pod/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7 Stopping container payload-pre-processing 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-a90958fa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Killing pod/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj Stopping container payload-processing 33m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-273105-openshift-default Scaled up replica set e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f from 0 to 1 33m Normal AddedLabel gateway/e2e-shared-b-w3-273105 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-273105 33m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f Created pod: e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld 33m Normal AddedInterface pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Add eth0 [10.134.0.110/23] from ovn-kubernetes 33m Normal Killing pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 Stopping container istio-proxy 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-273105 Scaled up replica set payload-processing-e2e-shared-b-w3-273105-5d95d79756 from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-273105 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc from 0 to 1 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-273105-5d95d79756 Error creating: pods "payload-processing-e2e-shared-b-w3-273105-5d95d79756-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-273105: serviceaccount "payload-processing-e2e-shared-b-w3-273105" not found 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc Error creating: pods "payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-273105: serviceaccount "payload-processing-e2e-shared-b-w3-273105" not found 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-273105-5d95d79756 Created pod: payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc Created pod: payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x 33m Normal Created pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Container created 33m Normal Pulled pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Container created 33m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Add eth0 [10.134.0.111/23] from ovn-kubernetes 33m Normal Started pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Container started 33m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Add eth0 [10.134.0.112/23] from ovn-kubernetes 33m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Container started 32m Normal Killing pod/payload-processing-e2e-ait-b0171017-8fc874664-qpttr Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Stopping container payload-pre-processing 32m Normal Killing pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Stopping container istio-proxy 32m Normal AddedLabel gateway/e2e-shared-a-w0-ae3373 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-ae3373 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft 32m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled up replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 0 to 1 32m Warning Unhealthy pod/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp Readiness probe failed: timeout: failed to connect service "10.133.0.142:50052" within 1s: context deadline exceeded 32m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Add eth0 [10.134.0.114/23] from ovn-kubernetes 32m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-b2b596-openshift-default Scaled up replica set e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 from 1 to 4 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 32m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-ae3373 Scaled up replica set payload-processing-e2e-shared-a-w0-ae3373-656cd95b64 from 0 to 1 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w 32m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Add eth0 [10.132.0.100/23] from ovn-kubernetes 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Container created 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Container started 32m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Readiness probe failed: Get "http://10.134.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64 Created pod: payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn 32m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Readiness probe error: Get "http://10.134.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 32m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64 Error creating: pods "payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373: serviceaccount "payload-processing-e2e-shared-a-w0-ae3373" not found 32m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-ae3373 Scaled up replica set payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7 from 0 to 1 32m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7 Created pod: payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg 32m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7 Error creating: pods "payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373: serviceaccount "payload-processing-e2e-shared-a-w0-ae3373" not found 32m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Add eth0 [10.133.0.148/23] from ovn-kubernetes 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Container created 32m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Add eth0 [10.134.0.115/23] from ovn-kubernetes 32m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Container started 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Container started 32m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Add eth0 [10.133.0.150/23] from ovn-kubernetes 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Container created 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Container created 32m Normal Pulled pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Add eth0 [10.133.0.149/23] from ovn-kubernetes 32m Normal Started pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Container started 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Container started 32m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Container created 32m Warning Unhealthy pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Readiness probe failed: Get "http://10.134.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Warning ProbeError pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr Readiness probe error: Get "http://10.134.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 32m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Startup probe failed: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused... 32m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Startup probe error: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Startup probe failed: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused 32m Normal ScalingReplicaSet deployment/e2e-mt-fb0abbf7-openshift-default Scaled up replica set e2e-mt-fb0abbf7-openshift-default-cff7b9797 from 0 to 1 32m Normal AddedLabel gateway/e2e-mt-fb0abbf7 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-fb0abbf7 32m Normal SuccessfulCreate replicaset/e2e-mt-fb0abbf7-openshift-default-cff7b9797 Created pod: e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft 32m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Startup probe failed: Get "http://10.133.0.148:15021/healthz/ready": dial tcp 10.133.0.148:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Startup probe error: Get "http://10.133.0.148:15021/healthz/ready": dial tcp 10.133.0.148:15021: connect: connection refused... 32m Normal AddedInterface pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Add eth0 [10.133.0.151/23] from ovn-kubernetes 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Container started 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Container created 32m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 Created pod: payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw 32m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 Error creating: pods "payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-fb0abbf7: serviceaccount "payload-processing-e2e-mt-fb0abbf7" not found 32m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-fb0abbf7-65697d6b56 Created pod: payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 32m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-fb0abbf7 Scaled up replica set payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 from 0 to 1 32m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-fb0abbf7 Scaled up replica set payload-processing-e2e-mt-fb0abbf7-65697d6b56 from 0 to 1 32m Warning FailedCreate replicaset/payload-processing-e2e-mt-fb0abbf7-65697d6b56 Error creating: pods "payload-processing-e2e-mt-fb0abbf7-65697d6b56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-fb0abbf7: serviceaccount "payload-processing-e2e-mt-fb0abbf7" not found 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal AddedInterface pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Add eth0 [10.134.0.117/23] from ovn-kubernetes 32m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Startup probe error: Get "http://10.134.0.104:15021/healthz/ready": dial tcp 10.134.0.104:15021: connect: connection refused... 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Startup probe failed: Get "http://10.134.0.104:15021/healthz/ready": dial tcp 10.134.0.104:15021: connect: connection refused 32m Normal Pulled pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Normal Pulled pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Normal AddedInterface pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Add eth0 [10.134.0.116/23] from ovn-kubernetes 32m Normal Started pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Container started 32m Normal Created pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Container created 32m Normal Created pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Container created 32m Normal Started pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Container started 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default New size: 6; reason: 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Created pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq 32m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-b2b596-openshift-default Scaled up replica set e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 from 4 to 6 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Container created 32m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Add eth0 [10.133.0.152/23] from ovn-kubernetes 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Container created 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Container started 32m Normal AddedInterface pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Add eth0 [10.132.0.103/23] from ovn-kubernetes 32m Normal Pulled pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Container started 31m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Startup probe failed: Get "http://10.132.0.103:15021/healthz/ready": dial tcp 10.132.0.103:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Startup probe error: Get "http://10.132.0.103:15021/healthz/ready": dial tcp 10.132.0.103:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Startup probe failed: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Startup probe error: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused... 31m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt 31m Normal AddedLabel gateway/e2e-shared-b-w0-e5d168 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-e5d168 31m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e5d168-openshift-default Scaled up replica set e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff from 0 to 1 31m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Add eth0 [10.134.0.119/23] from ovn-kubernetes 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6 Error creating: pods "payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168: serviceaccount "payload-processing-e2e-shared-b-w0-e5d168" not found 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6 Created pod: payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7 Error creating: pods "payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168: serviceaccount "payload-processing-e2e-shared-b-w0-e5d168" not found 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7 Created pod: payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-e5d168 Scaled up replica set payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6 from 0 to 1 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-e5d168 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7 from 0 to 1 31m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Container started 31m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Add eth0 [10.132.0.104/23] from ovn-kubernetes 31m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Container created 31m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Add eth0 [10.133.0.153/23] from ovn-kubernetes 31m Normal Pulled pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Created pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Container created 31m Normal Started pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Container started 31m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Readiness probe error: Get "http://10.134.0.114:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 31m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Readiness probe failed: Get "http://10.134.0.114:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal Created pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Container created 31m Normal Pulled pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Started pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Container started 31m Normal ScalingReplicaSet deployment/e2e-mt-9af5623f-openshift-default Scaled up replica set e2e-mt-9af5623f-openshift-default-66d4dbc5bf from 0 to 1 31m Normal AddedLabel gateway/e2e-mt-9af5623f Added label istio.io/rev=openshift-gateway to gateway e2e-mt-9af5623f 31m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr 31m Warning ProbeError pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Startup probe error: Get "http://10.133.0.151:15021/healthz/ready": dial tcp 10.133.0.151:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Startup probe failed: Get "http://10.133.0.151:15021/healthz/ready": dial tcp 10.133.0.151:15021: connect: connection refused 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-fb0abbf7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-fb0abbf7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Container created 31m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Container started 31m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Startup probe error: Get "http://10.134.0.114:15021/healthz/ready": dial tcp 10.134.0.114:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Startup probe failed: Get "http://10.134.0.114:15021/healthz/ready": dial tcp 10.134.0.114:15021: connect: connection refused 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Readiness probe error: Get "http://10.134.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 30m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Readiness probe failed: Get "http://10.134.0.119:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-fb0abbf7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-fb0abbf7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled up replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 1 to 4 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr 30m Normal SuccessfulCreate replicaset/e2e-api-a-f8709f-openshift-default-5579d88b9 Created pod: e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Add eth0 [10.132.0.111/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Add eth0 [10.133.0.154/23] from ovn-kubernetes 30m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Container started 30m Normal AddedLabel gateway/e2e-api-a-f8709f Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-f8709f 30m Normal ScalingReplicaSet deployment/e2e-api-a-f8709f-openshift-default Scaled up replica set e2e-api-a-f8709f-openshift-default-5579d88b9 from 0 to 1 30m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Container started 30m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Container created 30m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Container created 30m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Add eth0 [10.133.0.155/23] from ovn-kubernetes 30m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e5d168-openshift-default Scaled up replica set e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff from 1 to 4 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 9; reason: All metrics below target 30m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 10 to 9 30m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 30m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Container created 30m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Container created 30m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Startup probe error: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Startup probe failed: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused 30m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-f8709f Scaled up replica set payload-pre-processing-e2e-api-a-f8709f-8676d4ff from 0 to 1 30m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Container started 30m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-f8709f-8676d4ff Error creating: pods "payload-pre-processing-e2e-api-a-f8709f-8676d4ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f8709f: serviceaccount "payload-processing-e2e-api-a-f8709f" not found 30m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Startup probe error: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused... 30m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Warning FailedCreate replicaset/payload-processing-e2e-api-a-f8709f-b4dfdd89d Error creating: pods "payload-processing-e2e-api-a-f8709f-b4dfdd89d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f8709f: serviceaccount "payload-processing-e2e-api-a-f8709f" not found 30m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Container started 30m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Startup probe failed: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused 30m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Add eth0 [10.132.0.112/23] from ovn-kubernetes 30m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Add eth0 [10.133.0.157/23] from ovn-kubernetes 30m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-f8709f Scaled up replica set payload-processing-e2e-api-a-f8709f-b4dfdd89d from 0 to 1 30m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-f8709f-b4dfdd89d Created pod: payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg 30m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Container created 30m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Container started 30m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-f8709f-8676d4ff Created pod: payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq 30m Warning NodeNotReady pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Node is not ready 30m Normal Started pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Container started 30m Warning NodeNotReady pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Node is not ready 30m Warning NodeNotReady pod/payload-processing-5f78687d5b-j6n87 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Node is not ready 30m Warning NodeNotReady pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Node is not ready 30m Warning NodeNotReady pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Node is not ready 30m Warning NodeNotReady pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Node is not ready 30m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Add eth0 [10.133.0.158/23] from ovn-kubernetes 30m Normal Pulled pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal Created pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Container created 30m Warning NodeNotReady pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Node is not ready 30m Warning NodeNotReady pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Node is not ready 30m Warning NodeNotReady pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Node is not ready 30m Warning NodeNotReady pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Node is not ready 30m Warning NodeNotReady pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Node is not ready 30m Warning NodeNotReady pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Node is not ready 30m Warning NodeNotReady pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Node is not ready 30m Warning NodeNotReady pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Node is not ready 30m Normal Started pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Container started 30m Normal Created pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Container created 30m Warning NodeNotReady pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Node is not ready 30m Warning NodeNotReady pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Node is not ready 30m Warning NodeNotReady pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Node is not ready 30m Warning NodeNotReady pod/maas-default-gateway-openshift-default-77f556d647-4bknm Node is not ready 30m Warning NodeNotReady pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Node is not ready 30m Normal Pulled pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Add eth0 [10.133.0.159/23] from ovn-kubernetes 30m Warning NodeNotReady pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Node is not ready 30m Warning NodeNotReady pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Node is not ready 30m Warning NodeNotReady pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Node is not ready 30m Warning NodeNotReady pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Node is not ready 30m Warning NodeNotReady pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Node is not ready 30m Warning NodeNotReady pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Node is not ready 30m Warning NodeNotReady pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Node is not ready 30m Warning NodeNotReady pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Node is not ready 30m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Stopping container istio-proxy 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Add eth0 [10.134.0.121/23] from ovn-kubernetes 30m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Add eth0 [10.134.0.122/23] from ovn-kubernetes 30m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Add eth0 [10.134.0.120/23] from ovn-kubernetes 30m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Startup probe error: Get "http://10.134.0.122:15021/healthz/ready": dial tcp 10.134.0.122:15021: connect: connection refused... 30m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Startup probe error: Get "http://10.132.0.112:15021/healthz/ready": dial tcp 10.132.0.112:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Startup probe failed: Get "http://10.132.0.112:15021/healthz/ready": dial tcp 10.132.0.112:15021: connect: connection refused 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-273105-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Startup probe failed: Get "http://10.134.0.122:15021/healthz/ready": dial tcp 10.134.0.122:15021: connect: connection refused 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-273105-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Startup probe error: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Startup probe failed: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused 30m Normal TaintManagerEviction pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Cancelling deletion of Pod openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d 30m Normal TaintManagerEviction pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Cancelling deletion of Pod openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc 30m Normal TaintManagerEviction pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Cancelling deletion of Pod openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk 30m Normal TaintManagerEviction pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Cancelling deletion of Pod openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 30m Normal TaintManagerEviction pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Cancelling deletion of Pod openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft 30m Normal TaintManagerEviction pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Cancelling deletion of Pod openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw 30m Normal TaintManagerEviction pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Cancelling deletion of Pod openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 30m Normal TaintManagerEviction pod/payload-processing-5f78687d5b-j6n87 Cancelling deletion of Pod openshift-ingress/payload-processing-5f78687d5b-j6n87 30m Normal TaintManagerEviction pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Cancelling deletion of Pod openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr 30m Normal TaintManagerEviction pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Cancelling deletion of Pod openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 30m Normal TaintManagerEviction pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Cancelling deletion of Pod openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld 30m Normal TaintManagerEviction pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq Cancelling deletion of Pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq 30m Normal TaintManagerEviction pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Cancelling deletion of Pod openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx 30m Normal TaintManagerEviction pod/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn Cancelling deletion of Pod openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn 30m Normal TaintManagerEviction pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Cancelling deletion of Pod openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 30m Normal TaintManagerEviction pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Cancelling deletion of Pod openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff 30m Normal TaintManagerEviction pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Cancelling deletion of Pod openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt 30m Normal TaintManagerEviction pod/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd Cancelling deletion of Pod openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd 30m Normal TaintManagerEviction pod/maas-default-gateway-openshift-default-77f556d647-4bknm Cancelling deletion of Pod openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm 30m Normal TaintManagerEviction pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Cancelling deletion of Pod openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 30m Normal TaintManagerEviction pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p Cancelling deletion of Pod openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p 30m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Startup probe failed: Get "http://10.134.0.119:15021/healthz/ready": dial tcp 10.134.0.119:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Startup probe error: Get "http://10.134.0.119:15021/healthz/ready": dial tcp 10.134.0.119:15021: connect: connection refused... 30m Normal TaintManagerEviction pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Cancelling deletion of Pod openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7 30m Normal TaintManagerEviction pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Cancelling deletion of Pod openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f 30m Normal TaintManagerEviction pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Cancelling deletion of Pod openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 30m Normal TaintManagerEviction pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Cancelling deletion of Pod openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 30m Normal TaintManagerEviction pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Cancelling deletion of Pod openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv 30m Normal TaintManagerEviction pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Cancelling deletion of Pod openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr 30m Normal TaintManagerEviction pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Cancelling deletion of Pod openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7 30m Normal TaintManagerEviction pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Cancelling deletion of Pod openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx 30m Normal TaintManagerEviction pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Cancelling deletion of Pod openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z 30m Normal TaintManagerEviction pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Cancelling deletion of Pod openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz 30m Normal TaintManagerEviction pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Cancelling deletion of Pod openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 30m Normal TaintManagerEviction pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Cancelling deletion of Pod openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr 30m Normal TaintManagerEviction pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Cancelling deletion of Pod openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg 30m Normal TaintManagerEviction pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Cancelling deletion of Pod openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv 30m Normal TaintManagerEviction pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Cancelling deletion of Pod openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w 30m Normal TaintManagerEviction pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Cancelling deletion of Pod openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq 30m Normal TaintManagerEviction pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Cancelling deletion of Pod openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 30m Normal TaintManagerEviction pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Cancelling deletion of Pod openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 30m Normal TaintManagerEviction pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Cancelling deletion of Pod openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x 30m Normal TaintManagerEviction pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Cancelling deletion of Pod openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d 30m Normal TaintManagerEviction pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Cancelling deletion of Pod openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled up replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 4 to 8 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 8; reason: 30m Normal TaintManagerEviction pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Cancelling deletion of Pod openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm 30m Normal TaintManagerEviction pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Cancelling deletion of Pod openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw 30m Normal TaintManagerEviction pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Cancelling deletion of Pod openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Add eth0 [10.132.0.113/23] from ovn-kubernetes 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Add eth0 [10.134.0.123/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Container started 30m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Container created 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Add eth0 [10.133.0.161/23] from ovn-kubernetes 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Add eth0 [10.133.0.160/23] from ovn-kubernetes 30m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5 Created pod: payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd 30m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5 Error creating: pods "payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-9af5623f: serviceaccount "payload-processing-e2e-mt-9af5623f" not found 30m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Startup probe failed: Get "http://10.134.0.123:15021/healthz/ready": dial tcp 10.134.0.123:15021: connect: connection refused 30m Warning FailedCreate replicaset/payload-processing-e2e-mt-9af5623f-5854fb7bfb Error creating: pods "payload-processing-e2e-mt-9af5623f-5854fb7bfb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-9af5623f: serviceaccount "payload-processing-e2e-mt-9af5623f" not found 30m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Startup probe error: Get "http://10.134.0.123:15021/healthz/ready": dial tcp 10.134.0.123:15021: connect: connection refused... 30m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-9af5623f Scaled up replica set payload-processing-e2e-mt-9af5623f-5854fb7bfb from 0 to 1 30m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-9af5623f Scaled up replica set payload-pre-processing-e2e-mt-9af5623f-68cf6685c5 from 0 to 1 30m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-9af5623f-5854fb7bfb Created pod: payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 30m Normal AddedInterface pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Add eth0 [10.134.0.124/23] from ovn-kubernetes 30m Normal Started pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Container started 30m Normal Created pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Container created 30m Normal Pulled pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Add eth0 [10.134.0.125/23] from ovn-kubernetes 30m Normal Pulled pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal Created pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Container created 30m Normal Started pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Container started 30m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr Stopping container istio-proxy 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 6; reason: All metrics below target 30m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 9 to 6 30m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb 30m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 30m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr 30m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb Readiness probe failed: HTTP probe failed with statuscode: 503 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 10; reason: 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled up replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 8 to 10 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g 30m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Add eth0 [10.134.0.126/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Container created 30m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Container started 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Startup probe failed: Get "http://10.134.0.126:15021/healthz/ready": dial tcp 10.134.0.126:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Startup probe error: Get "http://10.134.0.126:15021/healthz/ready": dial tcp 10.134.0.126:15021: connect: connection refused... 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Container created 29m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Container started 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d 29m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn 29m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Add eth0 [10.133.0.163/23] from ovn-kubernetes 29m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Add eth0 [10.134.0.127/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e5d168-openshift-default Scaled up replica set e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff from 4 to 8 29m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Container created 29m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Container started 29m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Add eth0 [10.133.0.162/23] from ovn-kubernetes 29m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Container started 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 29m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Container created 29m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Startup probe error: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused... 29m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Startup probe error: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Startup probe failed: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused 29m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Startup probe failed: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused 29m Normal ScalingReplicaSet deployment/e2e-api-a-295497-openshift-default Scaled up replica set e2e-api-a-295497-openshift-default-58f5f7b8cc from 0 to 1 29m Normal AddedLabel gateway/e2e-api-a-295497 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-295497 29m Normal SuccessfulCreate replicaset/e2e-api-a-295497-openshift-default-58f5f7b8cc Created pod: e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm 29m Normal AddedInterface pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Add eth0 [10.134.0.129/23] from ovn-kubernetes 29m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Startup probe failed: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Startup probe error: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused... 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-295497-7878b9c7d6 Created pod: payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-295497 Scaled up replica set payload-pre-processing-e2e-api-a-295497-84b9b64fff from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-295497-84b9b64fff Created pod: payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 29m Warning FailedCreate replicaset/payload-processing-e2e-api-a-295497-7878b9c7d6 Error creating: pods "payload-processing-e2e-api-a-295497-7878b9c7d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-295497: serviceaccount "payload-processing-e2e-api-a-295497" not found 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-295497-84b9b64fff Error creating: pods "payload-pre-processing-e2e-api-a-295497-84b9b64fff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-295497: serviceaccount "payload-processing-e2e-api-a-295497" not found 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-295497 Scaled up replica set payload-processing-e2e-api-a-295497-7878b9c7d6 from 0 to 1 29m Normal Created pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Container created 29m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Add eth0 [10.134.0.130/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e5d168-openshift-default Scaled up replica set e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff from 8 to 9 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default New size: 9; reason: All metrics below target 29m Normal Pulled pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Add eth0 [10.133.0.164/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Created pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl 29m Normal Started pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Container started 29m Normal Pulled pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Container created 29m Normal Started pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Container started 29m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Add eth0 [10.132.0.114/23] from ovn-kubernetes 29m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Add eth0 [10.132.0.115/23] from ovn-kubernetes 29m Normal AddedInterface pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Add eth0 [10.132.0.117/23] from ovn-kubernetes 29m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Container created 29m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Container started 29m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Container created 29m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Container started 29m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Container created 29m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Container started 29m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Startup probe error: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Startup probe failed: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Startup probe error: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Startup probe failed: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused 29m Warning Unhealthy pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Startup probe failed: Get "http://10.132.0.117:15021/healthz/ready": dial tcp 10.132.0.117:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Startup probe error: Get "http://10.132.0.117:15021/healthz/ready": dial tcp 10.132.0.117:15021: connect: connection refused... 29m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled down replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 10 to 7 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 6; reason: All metrics below target 29m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled down replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 10 to 6 29m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 7; reason: All metrics below target 29m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled down replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 10 to 7 29m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 Stopping container istio-proxy 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 7; reason: All metrics below target 29m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 29m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw 29m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 29m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 29m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 Stopping container istio-proxy 29m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 Stopping container istio-proxy 29m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 Stopping container istio-proxy 29m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk 29m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj 29m Normal SuccessfulCreate replicaset/e2e-api-a-9066aa-openshift-default-7b8fff5d7 Created pod: e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 29m Normal AddedLabel gateway/e2e-api-a-9066aa Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-9066aa 29m Normal ScalingReplicaSet deployment/e2e-api-a-9066aa-openshift-default Scaled up replica set e2e-api-a-9066aa-openshift-default-7b8fff5d7 from 0 to 1 29m Normal AddedInterface pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Add eth0 [10.133.0.166/23] from ovn-kubernetes 29m Normal Killing pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Stopping container payload-processing 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-9066aa Scaled up replica set payload-processing-e2e-api-a-9066aa-c9bc596dd from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-9066aa Scaled up replica set payload-pre-processing-e2e-api-a-9066aa-cf84589ff from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-9066aa-c9bc596dd Created pod: payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-9066aa-cf84589ff Error creating: pods "payload-pre-processing-e2e-api-a-9066aa-cf84589ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-9066aa: serviceaccount "payload-processing-e2e-api-a-9066aa" not found 29m Warning FailedCreate replicaset/payload-processing-e2e-api-a-9066aa-c9bc596dd Error creating: pods "payload-processing-e2e-api-a-9066aa-c9bc596dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-9066aa: serviceaccount "payload-processing-e2e-api-a-9066aa" not found 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-9066aa-cf84589ff Created pod: payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt 29m Normal Pulled pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Add eth0 [10.133.0.168/23] from ovn-kubernetes 29m Normal Started pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Container started 29m Normal Created pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Container created 29m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Add eth0 [10.133.0.167/23] from ovn-kubernetes 29m Normal Created pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Container created 29m Normal Pulled pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Container started 29m Normal SuccessfulCreate replicaset/e2e-api-b-18cd5a-openshift-default-794dd87655 Created pod: e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 29m Normal ScalingReplicaSet deployment/e2e-api-b-18cd5a-openshift-default Scaled up replica set e2e-api-b-18cd5a-openshift-default-794dd87655 from 0 to 1 29m Normal AddedLabel gateway/e2e-api-b-18cd5a Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-18cd5a 29m Normal AddedInterface pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Add eth0 [10.133.0.169/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-18cd5a-57c7948996 Created pod: payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-18cd5a-76676f5699 Created pod: payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk 29m Warning FailedCreate replicaset/payload-processing-e2e-api-b-18cd5a-57c7948996 Error creating: pods "payload-processing-e2e-api-b-18cd5a-57c7948996-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-18cd5a: serviceaccount "payload-processing-e2e-api-b-18cd5a" not found 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-18cd5a Scaled up replica set payload-processing-e2e-api-b-18cd5a-57c7948996 from 0 to 1 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-18cd5a-76676f5699 Error creating: pods "payload-pre-processing-e2e-api-b-18cd5a-76676f5699-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-18cd5a: serviceaccount "payload-processing-e2e-api-b-18cd5a" not found 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-18cd5a Scaled up replica set payload-pre-processing-e2e-api-b-18cd5a-76676f5699 from 0 to 1 29m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Add eth0 [10.133.0.171/23] from ovn-kubernetes 29m Normal Pulled pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Add eth0 [10.133.0.170/23] from ovn-kubernetes 29m Normal Created pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Container created 29m Normal Created pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Container created 29m Normal Pulled pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Container started 29m Normal Started pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Container started 28m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-c22995-openshift-default Scaled up replica set e2e-shared-a-w4-c22995-openshift-default-769448f675 from 0 to 1 28m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-c22995-openshift-default-769448f675 Created pod: e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 28m Normal AddedLabel gateway/e2e-shared-a-w4-c22995 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w4-c22995 28m Normal AddedInterface pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Add eth0 [10.133.0.172/23] from ovn-kubernetes 28m Warning Unhealthy pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Warning Unhealthy pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled down replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 7 to 6 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 7; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled down replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 7 to 6 28m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 28m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 6; reason: All metrics below target 28m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 28m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc 28m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled down replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 10 to 7 28m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 6; reason: All metrics below target 28m Warning Unhealthy pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Readiness probe failed: timeout: failed to connect service "10.134.0.125:50052" within 1s: context deadline exceeded 28m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-mt-9af5623f-openshift-default Scaled up replica set e2e-mt-9af5623f-openshift-default-66d4dbc5bf from 1 to 4 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Container started 28m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Add eth0 [10.132.0.121/23] from ovn-kubernetes 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Add eth0 [10.133.0.175/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Container created 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Container started 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Add eth0 [10.133.0.174/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Container created 28m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe failed: HTTP probe failed with statuscode: 503 28m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Deleted pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-fdf4da-openshift-default New size: 6; reason: All metrics below target 28m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-worker-w5-fdf4da-openshift-default Scaled down replica set e2e-worker-w5-fdf4da-openshift-default-76967bcf57 from 10 to 6 28m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Deleted pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj 28m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm Stopping container istio-proxy 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 6; reason: All metrics below target 28m Normal SuccessfulDelete replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Deleted pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 28m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd 28m Normal SuccessfulDelete replicaset/e2e-worker-w5-fdf4da-openshift-default-76967bcf57 Deleted pod: e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm 28m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled down replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 7 to 6 28m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w 28m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 6 to 5 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 5; reason: All metrics below target 28m Normal SuccessfulCreate replicaset/e2e-api-b-550276-openshift-default-5496d6dbc7 Created pod: e2e-api-b-550276-openshift-default-5496d6dbc7-45knb 28m Normal ScalingReplicaSet deployment/e2e-api-b-550276-openshift-default Scaled up replica set e2e-api-b-550276-openshift-default-5496d6dbc7 from 0 to 1 28m Normal AddedLabel gateway/e2e-api-b-550276 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-550276 28m Normal AddedInterface pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Add eth0 [10.133.0.176/23] from ovn-kubernetes 28m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Startup probe failed: Get "http://10.133.0.174:15021/healthz/ready": dial tcp 10.133.0.174:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Startup probe failed: Get "http://10.133.0.175:15021/healthz/ready": dial tcp 10.133.0.175:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Startup probe failed: Get "http://10.132.0.121:15021/healthz/ready": dial tcp 10.132.0.121:15021: connect: connection refused 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Startup probe error: Get "http://10.133.0.174:15021/healthz/ready": dial tcp 10.133.0.174:15021: connect: connection refused... 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s 28m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Startup probe error: Get "http://10.132.0.121:15021/healthz/ready": dial tcp 10.132.0.121:15021: connect: connection refused... 28m Normal ScalingReplicaSet deployment/e2e-mt-9af5623f-openshift-default Scaled up replica set e2e-mt-9af5623f-openshift-default-66d4dbc5bf from 4 to 8 28m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Liveness probe failed: dial tcp 10.134.0.105:50052: i/o timeout 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Startup probe error: Get "http://10.133.0.175:15021/healthz/ready": dial tcp 10.133.0.175:15021: connect: connection refused... 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Container created 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Add eth0 [10.133.0.179/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Container created 28m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Readiness probe failed: Get "http://10.134.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Add eth0 [10.132.0.123/23] from ovn-kubernetes 28m Warning Unhealthy pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Liveness probe failed: dial tcp 10.134.0.106:50052: i/o timeout 28m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Readiness probe error: Get "http://10.134.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Add eth0 [10.133.0.177/23] from ovn-kubernetes 28m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 Readiness probe error: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Readiness probe failed: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Readiness probe failed: timeout: failed to connect service "10.134.0.61:50052" within 1s: context deadline exceeded 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Readiness probe failed: timeout: failed to connect service "10.134.0.62:50052" within 1s: context deadline exceeded 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Container created 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Container started 28m Warning Unhealthy pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Readiness probe failed: timeout: failed to connect service "10.134.0.130:50052" within 1s: context deadline exceeded 28m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Readiness probe error: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Container started 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Container started 28m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Readiness probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 28m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc Readiness probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 28m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Add eth0 [10.133.0.178/23] from ovn-kubernetes 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Readiness probe failed: Get "http://10.134.0.120:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Readiness probe error: Get "http://10.134.0.120:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 28m Warning Unhealthy pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw Readiness probe failed: timeout: failed to connect service "10.134.0.116:50052" within 1s: context deadline exceeded 28m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Readiness probe failed: timeout: failed to connect service "10.134.0.64:50052" within 1s: context deadline exceeded 28m Warning Unhealthy pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29 Readiness probe failed: timeout: failed to connect service "10.134.0.117:50052" within 1s: context deadline exceeded 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Startup probe error: Get "http://10.133.0.178:15021/healthz/ready": dial tcp 10.133.0.178:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Startup probe failed: Get "http://10.133.0.178:15021/healthz/ready": dial tcp 10.133.0.178:15021: connect: connection refused 28m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx Readiness probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Startup probe failed: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Startup probe error: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Startup probe failed: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Startup probe error: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Startup probe failed: Get "http://10.133.0.177:15021/healthz/ready": dial tcp 10.133.0.177:15021: connect: connection refused 28m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Startup probe error: Get "http://10.133.0.177:15021/healthz/ready": dial tcp 10.133.0.177:15021: connect: connection refused... 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w4-c22995 Scaled up replica set payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7 from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6 Created pod: payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w4-c22995 Scaled up replica set payload-processing-e2e-shared-a-w4-c22995-74cf4888c6 from 0 to 1 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7 Error creating: pods "payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-c22995: serviceaccount "payload-processing-e2e-shared-a-w4-c22995" not found 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7 Created pod: payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm 28m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6 Error creating: pods "payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-c22995: serviceaccount "payload-processing-e2e-shared-a-w4-c22995" not found 28m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default Scaled down replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 6 to 5 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 5; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default Scaled down replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 6 to 5 28m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g 28m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w 28m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 28m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled down replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 6 to 5 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 5; reason: All metrics below target 28m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w Stopping container istio-proxy 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 5; reason: All metrics below target 28m Normal Pulled pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Add eth0 [10.133.0.182/23] from ovn-kubernetes 28m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz 28m Normal Started pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Container started 28m Normal Created pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Container created 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 5; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled down replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 6 to 5 28m Normal Started pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Container started 28m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz Stopping container istio-proxy 28m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Add eth0 [10.133.0.181/23] from ovn-kubernetes 28m Normal Created pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Container created 28m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-550276 Scaled up replica set payload-pre-processing-e2e-api-b-550276-849c97f58c from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-550276-849c97f58c Created pod: payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-550276-849c97f58c Error creating: pods "payload-pre-processing-e2e-api-b-550276-849c97f58c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-550276: serviceaccount "payload-processing-e2e-api-b-550276" not found 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-550276 Scaled up replica set payload-processing-e2e-api-b-550276-94746b766 from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-550276-94746b766 Created pod: payload-processing-e2e-api-b-550276-94746b766-wzfqv 28m Warning FailedCreate replicaset/payload-processing-e2e-api-b-550276-94746b766 Error creating: pods "payload-processing-e2e-api-b-550276-94746b766-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-550276: serviceaccount "payload-processing-e2e-api-b-550276" not found 28m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Add eth0 [10.134.0.131/23] from ovn-kubernetes 28m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Add eth0 [10.134.0.132/23] from ovn-kubernetes 28m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Container created 28m Normal Pulled pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal Created pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Container created 28m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Container started 28m Normal Pulled pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal Started pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Container started 28m Normal Created pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Container created 28m Normal Started pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Container started 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-fb0abbf7 Scaled up replica set payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 from 0 to 1 28m Warning FailedCreate replicaset/payload-processing-e2e-mt-fb0abbf7-65697d6b56 Error creating: pods "payload-processing-e2e-mt-fb0abbf7-65697d6b56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-fb0abbf7: serviceaccount "payload-processing-e2e-mt-fb0abbf7" not found 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 Created pod: payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587 Error creating: pods "payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-fb0abbf7: serviceaccount "payload-processing-e2e-mt-fb0abbf7" not found 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-fb0abbf7 Scaled up replica set payload-processing-e2e-mt-fb0abbf7-65697d6b56 from 0 to 1 28m Normal Pulled pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-fb0abbf7-65697d6b56 Created pod: payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm 28m Normal Started pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Container started 28m Normal Created pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Container created 28m Normal AddedInterface pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Add eth0 [10.134.0.133/23] from ovn-kubernetes 28m Normal AddedInterface pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Add eth0 [10.134.0.134/23] from ovn-kubernetes 28m Normal Started pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Container started 28m Normal Created pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Container created 28m Normal Pulled pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal SuccessfulCreate replicaset/e2e-api-b-227c01-openshift-default-7987fbf485 Created pod: e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p 28m Normal AddedLabel gateway/e2e-api-b-227c01 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-227c01 28m Normal ScalingReplicaSet deployment/e2e-api-b-227c01-openshift-default Scaled up replica set e2e-api-b-227c01-openshift-default-7987fbf485 from 0 to 1 28m Normal AddedInterface pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Add eth0 [10.134.0.135/23] from ovn-kubernetes 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Deleted pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx 28m Normal Killing pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Stopping container payload-processing 28m Normal ScalingReplicaSet deployment/e2e-worker-w6-ef13cc-openshift-default Scaled down replica set e2e-worker-w6-ef13cc-openshift-default-5d997588b7 from 10 to 5 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-ef13cc-openshift-default New size: 5; reason: All metrics below target 28m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Deleted pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Deleted pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 28m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Deleted pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt 28m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-ef13cc-openshift-default-5d997588b7 Deleted pod: e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr 28m Normal Killing pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Stopping container payload-pre-processing 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-550276-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-550276-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-227c01-cb946789d Created pod: payload-processing-e2e-api-b-227c01-cb946789d-9nbjb 28m Warning FailedCreate replicaset/payload-processing-e2e-api-b-227c01-cb946789d Error creating: pods "payload-processing-e2e-api-b-227c01-cb946789d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-227c01: serviceaccount "payload-processing-e2e-api-b-227c01" not found 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-227c01 Scaled up replica set payload-processing-e2e-api-b-227c01-cb946789d from 0 to 1 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-227c01 Scaled up replica set payload-pre-processing-e2e-api-b-227c01-6b9459586c from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-227c01-6b9459586c Created pod: payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-227c01-6b9459586c Error creating: pods "payload-pre-processing-e2e-api-b-227c01-6b9459586c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-227c01: serviceaccount "payload-processing-e2e-api-b-227c01" not found 28m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Add eth0 [10.134.0.136/23] from ovn-kubernetes 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal Pulled pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Started pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Container started 27m Normal Pulled pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Add eth0 [10.134.0.137/23] from ovn-kubernetes 27m Normal Created pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Container created 27m Normal Created pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Container created 27m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Readiness probe failed: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 Readiness probe error: Get "http://10.133.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal Started pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Container started 27m Warning Unhealthy pod/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal SuccessfulCreate replicaset/e2e-mt-9af5623f-openshift-default-66d4dbc5bf Created pod: e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 27m Normal Killing pod/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq Stopping container payload-pre-processing 27m Normal ScalingReplicaSet deployment/e2e-mt-9af5623f-openshift-default Scaled up replica set e2e-mt-9af5623f-openshift-default-66d4dbc5bf from 8 to 9 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-9af5623f-openshift-default New size: 9; reason: All metrics below target 27m Normal Killing pod/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg Stopping container payload-processing 27m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Add eth0 [10.134.0.140/23] from ovn-kubernetes 27m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Container created 27m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Container started 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-f0837a-openshift-default-774944cc99 Created pod: e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-f0837a-openshift-default Scaled up replica set e2e-shared-b-w4-f0837a-openshift-default-774944cc99 from 0 to 1 27m Normal AddedLabel gateway/e2e-shared-b-w4-f0837a Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w4-f0837a 27m Normal AddedInterface pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Add eth0 [10.133.0.187/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Startup probe error: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Startup probe failed: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused 27m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 4; reason: All metrics below target 27m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q Stopping container istio-proxy 27m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 5 to 4 27m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh Stopping container istio-proxy 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-b2b596-openshift-default Scaled down replica set e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 from 6 to 4 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default New size: 4; reason: All metrics below target 27m Normal SuccessfulDelete replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Deleted pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq 27m Normal SuccessfulDelete replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Deleted pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh 27m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq Stopping container istio-proxy 27m Warning Unhealthy pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Readiness probe failed: Get "http://10.133.0.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning ProbeError pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Readiness probe error: Get "http://10.133.0.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c Created pod: payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w4-f0837a Scaled up replica set payload-processing-e2e-shared-b-w4-f0837a-594bb5fb from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb Created pod: payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c Error creating: pods "payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a: serviceaccount "payload-processing-e2e-shared-b-w4-f0837a" not found 27m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb Error creating: pods "payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a: serviceaccount "payload-processing-e2e-shared-b-w4-f0837a" not found 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w4-f0837a Scaled up replica set payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c from 0 to 1 27m Normal Started pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Add eth0 [10.133.0.188/23] from ovn-kubernetes 27m Normal Created pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Container created 27m Normal Pulled pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Container created 27m Normal AddedInterface pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Add eth0 [10.133.0.189/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Container started 27m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Container created 27m Normal Pulled pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Pulled pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Container started 27m Normal Created pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Container created 27m Normal Pulled pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Container started 27m Normal Started pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Container started 27m Normal Created pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Container created 27m Normal Pulled pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Container started 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-c22995-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal Created pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Container created 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-c22995-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-f0837a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-f0837a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning ProbeError pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Startup probe error: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Startup probe failed: Get "http://10.134.0.129:15021/healthz/ready": dial tcp 10.134.0.129:15021: connect: connection refused 27m Warning ProbeError pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Startup probe error: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Startup probe failed: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused 27m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: All metrics below target 27m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 4 to 3 27m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-2s5j4 27m Warning ProbeError pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Startup probe error: Get "http://10.133.0.172:15021/healthz/ready": dial tcp 10.133.0.172:15021: connect: connection refused... 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-18cd5a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-18cd5a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning Unhealthy pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Startup probe failed: Get "http://10.133.0.172:15021/healthz/ready": dial tcp 10.133.0.172:15021: connect: connection refused 27m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 Stopping container discovery 27m Warning ProbeError pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Startup probe error: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Startup probe failed: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 4; reason: All metrics below target 27m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled down replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 5 to 4 27m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p 27m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Stopping container istio-proxy 27m Normal Killing pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Stopping container istio-proxy 27m Normal Killing pod/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6 Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Stopping container payload-pre-processing 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Readiness probe error: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-f0837a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-f0837a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning Unhealthy pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf Readiness probe failed: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning Unhealthy pod/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd Readiness probe failed: timeout: failed to connect service "10.134.0.124:50052" within 1s: context deadline exceeded 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-295497-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-295497-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f8709f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f8709f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv 27m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default Scaled down replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 5 to 4 27m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 27m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 Stopping container istio-proxy 27m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Readiness probe error: Get "http://10.132.0.123:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default New size: 4; reason: All metrics below target 27m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv Stopping container istio-proxy 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 4; reason: All metrics below target 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default New size: 4; reason: All metrics below target 27m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Readiness probe failed: Get "http://10.132.0.123:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q Stopping container istio-proxy 27m Warning BackOff pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr Back-off restarting failed container istio-proxy in pod e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr_openshift-ingress(fea5d15c-4957-49c0-9739-f5383e23f424) 27m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q 27m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Container created 27m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Container started 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-b2b596-openshift-default Scaled down replica set e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 from 4 to 3 27m Normal Created pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Container created 27m Normal Pulled pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal SuccessfulDelete replicaset/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7 Deleted pod: e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-b2b596-openshift-default New size: 3; reason: All metrics below target 27m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 Stopping container istio-proxy 27m Normal Started pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Container started 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Stopping container istio-proxy 27m Normal Killing pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf Stopping container istio-proxy 27m Warning Unhealthy pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Readiness probe failed: Get "http://10.133.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning ProbeError pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Readiness probe error: Get "http://10.133.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-c22995-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-c22995-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Normal Killing pod/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5 Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw Stopping container payload-processing 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-295497-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-295497-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f8709f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f8709f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-18cd5a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-18cd5a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Readiness probe error: Get "http://10.133.0.178:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg Readiness probe failed: Get "http://10.133.0.178:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Readiness probe error: Get "http://10.133.0.179:15021/healthz/ready": read tcp 10.133.0.2:42454->10.133.0.179:15021: read: connection reset by peer... 26m Warning ProbeError pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Readiness probe error: Get "http://10.133.0.177:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n Readiness probe failed: Get "http://10.133.0.179:15021/healthz/ready": read tcp 10.133.0.2:42454->10.133.0.179:15021: read: connection reset by peer 26m Warning Unhealthy pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s Readiness probe failed: Get "http://10.133.0.177:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal Created pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Container created 26m Normal Started pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Container started 26m Normal Pulled pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning Unhealthy pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Startup probe failed: Get "http://10.133.0.176:15021/healthz/ready": dial tcp 10.133.0.176:15021: connect: connection refused 26m Warning ProbeError pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Startup probe error: Get "http://10.133.0.176:15021/healthz/ready": dial tcp 10.133.0.176:15021: connect: connection refused... 26m Normal Killing pod/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm Stopping container payload-processing 26m Normal ScalingReplicaSet deployment/e2e-worker-w1-fb2fa0-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 from 4 to 3 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-fb2fa0-openshift-default (combined from similar events): New size: 3; reason: All metrics below target 26m Normal SuccessfulDelete replicaset/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5 Deleted pod: e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w 26m Normal Killing pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Stopping container istio-proxy 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-550276-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-550276-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning Unhealthy pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Readiness probe failed: HTTP probe failed with statuscode: 503 26m Warning ProbeError pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w Readiness probe error: HTTP probe failed with statuscode: 503... 26m Normal Created pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Container created 26m Normal Pulled pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Container started 26m Normal Killing pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Stopping container payload-processing 26m Normal Killing pod/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt Stopping container payload-pre-processing 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-227c01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-227c01-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 4 to 3 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 3; reason: All metrics below target 26m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 26m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 Stopping container istio-proxy 26m Warning Unhealthy pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused 26m Warning ProbeError pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused... 26m Warning Unhealthy pod/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk Readiness probe failed: timeout: failed to connect service "10.133.0.171:50052" within 1s: context deadline exceeded 26m Warning Unhealthy pod/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7 Readiness probe failed: timeout: failed to connect service "10.133.0.170:50052" within 1s: context deadline exceeded 26m Warning ProbeError pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Readiness probe error: Get "http://10.133.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Readiness probe failed: Get "http://10.133.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal SuccessfulCreate replicaset/e2e-inf-a-0f0c50-openshift-default-6f96d756dc Created pod: e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 26m Normal ScalingReplicaSet deployment/e2e-inf-a-0f0c50-openshift-default Scaled up replica set e2e-inf-a-0f0c50-openshift-default-6f96d756dc from 0 to 1 26m Normal AddedLabel gateway/e2e-inf-a-0f0c50 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-0f0c50 26m Normal AddedInterface pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Add eth0 [10.133.0.198/23] from ovn-kubernetes 26m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Readiness probe error: Get "http://10.133.0.23:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Readiness probe failed: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-wmxgt Readiness probe error: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Readiness probe failed: Get "http://10.133.0.23:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Normal Pulled pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Container created 26m Normal Started pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Container started 26m Normal ScalingReplicaSet deployment/e2e-worker-w0-6ce5eb-openshift-default Scaled down replica set e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 from 4 to 3 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-227c01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 26m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 Stopping container istio-proxy 26m Normal SuccessfulDelete replicaset/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9 Deleted pod: e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 26m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 Stopping container istio-proxy 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6ce5eb-openshift-default New size: 3; reason: All metrics below target 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-227c01-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-0f0c50 Scaled up replica set payload-pre-processing-e2e-inf-a-0f0c50-86786c47 from 0 to 1 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-0f0c50 Scaled up replica set payload-processing-e2e-inf-a-0f0c50-6f56666484 from 0 to 1 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-0f0c50-86786c47 Error creating: pods "payload-pre-processing-e2e-inf-a-0f0c50-86786c47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0f0c50: serviceaccount "payload-processing-e2e-inf-a-0f0c50" not found 26m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-0f0c50-6f56666484 Error creating: pods "payload-processing-e2e-inf-a-0f0c50-6f56666484-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0f0c50: serviceaccount "payload-processing-e2e-inf-a-0f0c50" not found 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-0f0c50-6f56666484 Created pod: payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-0f0c50-86786c47 Created pod: payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 26m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Add eth0 [10.133.0.199/23] from ovn-kubernetes 26m Normal Pulled pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Container created 26m Normal Created pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Container created 26m Normal Pulled pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Add eth0 [10.133.0.200/23] from ovn-kubernetes 26m Normal Started pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Container started 26m Normal Killing pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm Stopping container istio-proxy 26m Normal Started pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Container started 26m Warning ProbeError pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Startup probe error: Get "http://10.133.0.166:15021/healthz/ready": dial tcp 10.133.0.166:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Startup probe failed: Get "http://10.133.0.166:15021/healthz/ready": dial tcp 10.133.0.166:15021: connect: connection refused 25m Normal Killing pod/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786 Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-api-b-550276-94746b766-wzfqv Stopping container payload-processing 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-9066aa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-9066aa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled down replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 10 to 8 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 8; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Deleted pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj 25m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Stopping container istio-proxy 25m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj Stopping container istio-proxy 25m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Deleted pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv Readiness probe error: HTTP probe failed with statuscode: 503... 25m Normal Killing pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft Stopping container istio-proxy 25m Normal Killing pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 Stopping container istio-proxy 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-e5d168-openshift-default New size: 8; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff Deleted pod: e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e5d168-openshift-default Scaled down replica set e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff from 9 to 8 25m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn Stopping container istio-proxy 25m Normal Killing pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 Stopping container istio-proxy 25m Normal Killing pod/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4 Stopping container payload-pre-processing 25m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn Stopping container payload-processing 25m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled down replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 8 to 6 25m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Deleted pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g 25m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Stopping container istio-proxy 25m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z Stopping container istio-proxy 25m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c Deleted pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 6; reason: All metrics below target 25m Normal SuccessfulCreate replicaset/e2e-inf-b-69c44c-openshift-default-578bd575cf Created pod: e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn 25m Normal ScalingReplicaSet deployment/e2e-inf-b-69c44c-openshift-default Scaled up replica set e2e-inf-b-69c44c-openshift-default-578bd575cf from 0 to 1 25m Normal AddedLabel gateway/e2e-inf-b-69c44c Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-69c44c 25m Normal AddedInterface pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Add eth0 [10.134.0.142/23] from ovn-kubernetes 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Readiness probe error: Get "http://10.133.0.161:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Readiness probe error: Get "http://10.132.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Readiness probe failed: Get "http://10.133.0.161:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Readiness probe error: Get "http://10.133.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Readiness probe failed: Get "http://10.133.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Readiness probe failed: Get "http://10.132.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Container started 25m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Container created 25m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Container created 25m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Container started 25m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Container started 25m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Container started 25m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Container created 25m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Container created 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-69c44c Scaled up replica set payload-processing-e2e-inf-b-69c44c-684b86fb7b from 0 to 1 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc Error creating: pods "payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-69c44c: serviceaccount "payload-processing-e2e-inf-b-69c44c" not found 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc Created pod: payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-69c44c-684b86fb7b Created pod: payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 25m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-69c44c-684b86fb7b Error creating: pods "payload-processing-e2e-inf-b-69c44c-684b86fb7b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-69c44c: serviceaccount "payload-processing-e2e-inf-b-69c44c" not found 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-69c44c Scaled up replica set payload-pre-processing-e2e-inf-b-69c44c-544478dfdc from 0 to 1 25m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Startup probe failed: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Startup probe error: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused... 25m Normal Killing pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Stopping container istio-proxy 25m Normal Pulled pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Add eth0 [10.133.0.205/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Add eth0 [10.133.0.206/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Container created 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Startup probe error: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Startup probe failed: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused 24m Warning ProbeError pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Created pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Container created 24m Normal Started pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Container started 24m Normal Started pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Container started 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Startup probe error: Get "http://10.133.0.160:15021/healthz/ready": dial tcp 10.133.0.160:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Startup probe failed: Get "http://10.133.0.160:15021/healthz/ready": dial tcp 10.133.0.160:15021: connect: connection refused 24m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ae3373-openshift-default Scaled up replica set e2e-shared-a-w0-ae3373-openshift-default-78486566c from 6 to 10 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Startup probe failed: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused 24m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ae3373-openshift-default-78486566c (combined from similar events): Created pod: e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ae3373-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Startup probe error: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused... 24m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Container started 24m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Add eth0 [10.132.0.125/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Container started 24m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Container created 24m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Container created 24m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Container created 24m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Add eth0 [10.134.0.143/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Container created 24m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Add eth0 [10.133.0.208/23] from ovn-kubernetes 24m Normal AddedInterface pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Add eth0 [10.134.0.144/23] from ovn-kubernetes 24m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Container started 24m Normal Started pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Container started 24m Warning Unhealthy pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Readiness probe failed: Get "http://10.133.0.198:15021/healthz/ready": read tcp 10.133.0.2:37382->10.133.0.198:15021: read: connection reset by peer 24m Warning ProbeError pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Readiness probe error: Get "http://10.133.0.198:15021/healthz/ready": read tcp 10.133.0.2:37382->10.133.0.198:15021: read: connection reset by peer... 24m Normal Created pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Container created 24m Normal Pulled pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Container started 24m Normal Killing pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Stopping container payload-processing 24m Normal Killing pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Stopping container payload-pre-processing 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Startup probe error: Get "http://10.134.0.144:15021/healthz/ready": dial tcp 10.134.0.144:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Startup probe failed: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Startup probe failed: Get "http://10.134.0.143:15021/healthz/ready": dial tcp 10.134.0.143:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Startup probe error: Get "http://10.134.0.143:15021/healthz/ready": dial tcp 10.134.0.143:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Startup probe failed: Get "http://10.133.0.208:15021/healthz/ready": dial tcp 10.133.0.208:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Startup probe failed: Get "http://10.134.0.144:15021/healthz/ready": dial tcp 10.134.0.144:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Startup probe error: Get "http://10.133.0.208:15021/healthz/ready": dial tcp 10.133.0.208:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Startup probe error: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Startup probe error: Get "http://10.133.0.198:15021/healthz/ready": dial tcp 10.133.0.198:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Startup probe failed: Get "http://10.133.0.198:15021/healthz/ready": dial tcp 10.133.0.198:15021: connect: connection refused 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w Stopping container istio-proxy 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf Stopping container istio-proxy 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr Readiness probe error: HTTP probe failed with statuscode: 503... 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Stopping container istio-proxy 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-0f0c50-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-0f0c50-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Killing pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Stopping container istio-proxy 24m Normal ScalingReplicaSet deployment/e2e-inf-a-1b0848-openshift-default Scaled up replica set e2e-inf-a-1b0848-openshift-default-6874f48d45 from 0 to 1 24m Normal AddedLabel gateway/e2e-inf-a-1b0848 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-1b0848 24m Normal SuccessfulCreate replicaset/e2e-inf-a-1b0848-openshift-default-6874f48d45 Created pod: e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm 24m Normal AddedInterface pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Add eth0 [10.133.0.210/23] from ovn-kubernetes 24m Warning Unhealthy pod/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning ProbeError pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964 Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b Stopping container payload-processing 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-1b0848 Scaled up replica set payload-processing-e2e-inf-a-1b0848-79644c86fd from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-1b0848 Scaled up replica set payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc from 0 to 1 24m Normal AddedInterface pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Add eth0 [10.133.0.213/23] from ovn-kubernetes 24m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-1b0848-79644c86fd Error creating: pods "payload-processing-e2e-inf-a-1b0848-79644c86fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1b0848: serviceaccount "payload-processing-e2e-inf-a-1b0848" not found 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc Error creating: pods "payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1b0848: serviceaccount "payload-processing-e2e-inf-a-1b0848" not found 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-1b0848-79644c86fd Created pod: payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc Created pod: payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r 24m Normal Started pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Container started 24m Normal Started pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Container started 24m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Add eth0 [10.134.0.145/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Container created 24m Normal Pulled pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Container created 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-0f0c50-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-0f0c50-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Normal Started pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Container started 24m Normal Pulled pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 Stopping container istio-proxy 24m Normal Created pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Container created 24m Normal Killing pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 Stopping container istio-proxy 24m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5 Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4 Stopping container payload-processing 24m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Readiness probe failed: Get "http://10.134.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Readiness probe error: Get "http://10.134.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Readiness probe failed: Get "http://10.134.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Readiness probe error: Get "http://10.134.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Normal Killing pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Stopping container payload-processing 24m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr Stopping container istio-proxy 24m Normal Pulled pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Stopping container istio-proxy 23m Normal Started pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Container started 23m Normal Created pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Container created 23m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c Readiness probe error: HTTP probe failed with statuscode: 503... 23m Warning ProbeError pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Readiness probe error: HTTP probe failed with statuscode: 503... 23m Warning Unhealthy pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w Readiness probe failed: HTTP probe failed with statuscode: 503 23m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp Stopping container istio-proxy 23m Warning Unhealthy pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 23m Warning ProbeError pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg Readiness probe error: HTTP probe failed with statuscode: 503... 23m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm Stopping container payload-processing 23m Normal ScalingReplicaSet deployment/e2e-inf-b-c9a107-openshift-default Scaled up replica set e2e-inf-b-c9a107-openshift-default-fdbc7864c from 0 to 1 23m Normal AddedLabel gateway/e2e-inf-b-c9a107 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-c9a107 23m Normal SuccessfulCreate replicaset/e2e-inf-b-c9a107-openshift-default-fdbc7864c Created pod: e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 23m Normal AddedInterface pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Add eth0 [10.134.0.148/23] from ovn-kubernetes 23m Warning Unhealthy pod/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Readiness probe failed: Get "http://10.134.0.110:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 23m Warning ProbeError pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Readiness probe error: Get "http://10.134.0.110:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-c9a107 Scaled up replica set payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c Created pod: payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c Error creating: pods "payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-c9a107: serviceaccount "payload-processing-e2e-inf-b-c9a107" not found 23m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-c9a107-75cf5c66c6 Error creating: pods "payload-processing-e2e-inf-b-c9a107-75cf5c66c6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-c9a107: serviceaccount "payload-processing-e2e-inf-b-c9a107" not found 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-c9a107-75cf5c66c6 Created pod: payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-c9a107 Scaled up replica set payload-processing-e2e-inf-b-c9a107-75cf5c66c6 from 0 to 1 23m Normal Started pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Container started 23m Normal Started pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Container started 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-69c44c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal Pulled pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Container created 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-69c44c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Add eth0 [10.134.0.149/23] from ovn-kubernetes 23m Normal Pulled pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Container created 23m Normal AddedInterface pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Add eth0 [10.134.0.150/23] from ovn-kubernetes 23m Normal Created pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Container created 23m Normal Started pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Container started 23m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f Stopping container istio-proxy 23m Normal Pulled pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh Stopping container istio-proxy 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8b6cf1-openshift-default New size: 3; reason: All metrics below target 23m Normal ScalingReplicaSet deployment/e2e-worker-w3-8b6cf1-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w3-8b6cf1-openshift-default-855578f895 from 4 to 3 23m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m Stopping container istio-proxy 23m Normal SuccessfulDelete replicaset/e2e-worker-w3-8b6cf1-openshift-default-855578f895 Deleted pod: e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m 23m Warning ProbeError pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Startup probe error: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Startup probe failed: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused 23m Normal Killing pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Stopping container payload-pre-processing 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-273105-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal Killing pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Stopping container payload-processing 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-273105-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal Killing pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld Stopping container istio-proxy 23m Normal Pulled pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Started pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Container started 23m Normal Created pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Container created 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-69c44c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-69c44c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1b0848-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1b0848-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Warning Unhealthy pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Startup probe failed: Get "http://10.133.0.210:15021/healthz/ready": dial tcp 10.133.0.210:15021: connect: connection refused 23m Warning ProbeError pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Startup probe error: Get "http://10.133.0.210:15021/healthz/ready": dial tcp 10.133.0.210:15021: connect: connection refused... 23m Normal Killing pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Stopping container payload-processing 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-c9a107-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-c9a107-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1b0848-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1b0848-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Normal Pulled pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Container created 22m Normal Started pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Container started 22m Warning Unhealthy pod/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 22m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 22m Warning ProbeError pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Startup probe error: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Startup probe failed: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused 22m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k Stopping container istio-proxy 22m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst Stopping container istio-proxy 22m Normal Killing pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z Stopping container istio-proxy 22m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r Stopping container istio-proxy 22m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Stopping container istio-proxy 22m Normal Killing pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd Stopping container istio-proxy 22m Normal AddedInterface pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Add eth0 [10.134.0.156/23] from ovn-kubernetes 22m Normal Created pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Container created 22m Normal Started pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Container started 22m Normal SuccessfulCreate replicaset/e2e-inf-a-1b0848-openshift-default-6874f48d45 Created pod: e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj 22m Normal Pulled pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal ScalingReplicaSet deployment/e2e-inf-a-1b0848-openshift-default Scaled up replica set e2e-inf-a-1b0848-openshift-default-6874f48d45 from 1 to 2 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-1b0848-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 22m Warning Unhealthy pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Readiness probe failed: HTTP probe failed with statuscode: 503 22m Warning ProbeError pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning ProbeError pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Startup probe error: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Startup probe failed: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-c9a107-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-c9a107-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Normal ScalingReplicaSet deployment/e2e-worker-w4-4ef6ac-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 from 3 to 2 22m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Stopping container istio-proxy 22m Normal SuccessfulDelete replicaset/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799 Deleted pod: e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-4ef6ac-openshift-default New size: 2; reason: All metrics below target 22m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Readiness probe failed: HTTP probe failed with statuscode: 503 22m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf Readiness probe error: HTTP probe failed with statuscode: 503... 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-fde2c5-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-worker-w2-fde2c5-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 from 3 to 2 21m Normal SuccessfulDelete replicaset/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7 Deleted pod: e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 21m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 Stopping container istio-proxy 21m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 21m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-xttzf 21m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf Stopping container discovery 21m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 20m Normal Killing pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Stopping container payload-pre-processing 20m Normal Killing pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k Stopping container payload-processing 20m Normal Killing pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 Stopping container istio-proxy 20m Normal Killing pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj Stopping container istio-proxy 20m Normal Killing pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Stopping container istio-proxy 20m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm Readiness probe error: HTTP probe failed with statuscode: 503... 20m Normal Killing pod/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6 Stopping container payload-processing 20m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Stopping container payload-pre-processing 20m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r Readiness probe failed: timeout: failed to connect service "10.134.0.145:50052" within 1s: context deadline exceeded 20m Warning Unhealthy pod/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz Readiness probe failed: timeout: failed to connect service "10.134.0.150:50052" within 1s: context deadline exceeded 20m Normal Killing pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn Stopping container istio-proxy 19m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Stopping container istio-proxy 19m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning ProbeError pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 Readiness probe error: HTTP probe failed with statuscode: 503... 19m Normal Killing pod/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6 Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Stopping container payload-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79 Readiness probe failed: timeout: failed to connect service "10.134.0.149:50052" within 1s: context deadline exceeded 19m Normal Killing pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 Stopping container istio-proxy 19m Normal Killing pod/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh Stopping container payload-pre-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Started pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Container started 19m Normal Pulled pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Created pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Container created 19m Normal Killing pod/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Stopping container istio-proxy 19m Warning Unhealthy pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning ProbeError pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 Readiness probe error: HTTP probe failed with statuscode: 503... 19m Normal Killing pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Stopping container payload-processing 18m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-77f556d647 Deleted pod: maas-default-gateway-openshift-default-77f556d647-vhrw7 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-77f556d647 from 10 to 7 18m Normal Killing pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Stopping container istio-proxy 18m Normal Killing pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Stopping container istio-proxy 18m Normal Killing pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-77f556d647 Deleted pod: maas-default-gateway-openshift-default-77f556d647-b9qj7 18m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-77f556d647 Deleted pod: maas-default-gateway-openshift-default-77f556d647-bb5z2 18m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Stopping container istio-proxy 18m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Normal Killing pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z Stopping container istio-proxy 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Readiness probe error: Get "http://10.134.0.22:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 Readiness probe failed: Get "http://10.134.0.22:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: All metrics below target 18m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-77f556d647 Deleted pod: maas-default-gateway-openshift-default-77f556d647-cxtkt 18m Normal Killing pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-77f556d647 from 7 to 6 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-cxtkt Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Add eth0 [10.133.0.246/23] from ovn-kubernetes 18m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-77f556d647 from 6 to 10 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Add eth0 [10.134.0.159/23] from ovn-kubernetes 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Add eth0 [10.133.0.247/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-h2w59 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-f5zxr 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-zzlvr 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-77f556d647 Created pod: maas-default-gateway-openshift-default-77f556d647-kbsdt 18m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 18m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Container created 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Add eth0 [10.134.0.160/23] from ovn-kubernetes 18m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Container started 18m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Container started 18m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Container created 18m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Container started 18m Normal Started pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Container started 18m Normal Created pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Startup probe failed: Get "http://10.133.0.246:15021/healthz/ready": dial tcp 10.133.0.246:15021: connect: connection refused 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Startup probe error: Get "http://10.133.0.246:15021/healthz/ready": dial tcp 10.133.0.246:15021: connect: connection refused... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Startup probe failed: Get "http://10.133.0.247:15021/healthz/ready": dial tcp 10.133.0.247:15021: connect: connection refused 18m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp Stopping container discovery 18m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-t7ntp 18m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 18m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Startup probe error: Get "http://10.133.0.247:15021/healthz/ready": dial tcp 10.133.0.247:15021: connect: connection refused... 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Startup probe error: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Startup probe failed: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Startup probe error: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Startup probe failed: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Startup probe error: Get "http://10.133.0.246:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Startup probe failed: Get "http://10.133.0.247:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Startup probe error: Get "http://10.133.0.247:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Startup probe failed: Get "http://10.133.0.246:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Startup probe failed: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Startup probe failed: Get "http://10.134.0.159:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Startup probe error: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Startup probe error: Get "http://10.134.0.159:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Readiness probe failed: Get "http://10.133.0.246:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 13m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-zzlvr Readiness probe error: Get "http://10.133.0.246:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Readiness probe failed: Get "http://10.133.0.247:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 13m Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-kbsdt Readiness probe error: Get "http://10.133.0.247:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Normal AddedInterface pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Add eth0 [10.133.1.8/23] from ovn-kubernetes 12m Normal ScalingReplicaSet deployment/e2e-worker-single-main-f57fb7-openshift-default Scaled up replica set e2e-worker-single-main-f57fb7-openshift-default-5647675bf from 0 to 1 12m Normal SuccessfulCreate replicaset/e2e-worker-single-main-f57fb7-openshift-default-5647675bf Created pod: e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj 12m Normal AddedLabel gateway/e2e-worker-single-main-f57fb7 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-f57fb7 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-f57fb7 Scaled up replica set payload-processing-e2e-worker-single-main-f57fb7-75558bd49d from 0 to 1 12m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-f57fb7-75558bd49d Error creating: pods "payload-processing-e2e-worker-single-main-f57fb7-75558bd49d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7: serviceaccount "payload-processing-e2e-worker-single-main-f57fb7" not found 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-f57fb7-6b5844568c Error creating: pods "payload-pre-processing-e2e-worker-single-main-f57fb7-6b5844568c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7: serviceaccount "payload-processing-e2e-worker-single-main-f57fb7" not found 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-f57fb7-6b5844568c Created pod: payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-f57fb7-75558bd49d Created pod: payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-f57fb7 Scaled up replica set payload-pre-processing-e2e-worker-single-main-f57fb7-6b5844568c from 0 to 1 12m Normal Created pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Container created 12m Normal Created pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Container created 12m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal Pulled pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Add eth0 [10.133.1.11/23] from ovn-kubernetes 12m Normal Started pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Container started 12m Normal Started pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Container started 12m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Add eth0 [10.133.1.10/23] from ovn-kubernetes 12m Warning ProbeError pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Readiness probe error: Get "http://10.133.1.8:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Warning Unhealthy pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Readiness probe failed: Get "http://10.133.1.8:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 12m Normal Started pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Container started 12m Normal Pulled pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Container created 12m Warning Unhealthy pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Startup probe failed: Get "http://10.133.1.8:15021/healthz/ready": dial tcp 10.133.1.8:15021: connect: connection refused 12m Warning ProbeError pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Startup probe error: Get "http://10.133.1.8:15021/healthz/ready": dial tcp 10.133.1.8:15021: connect: connection refused... 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-f57fb7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-f57fb7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Normal Killing pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Stopping container payload-processing 11m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Stopping container payload-pre-processing 11m Normal Killing pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Stopping container istio-proxy 11m Warning Unhealthy pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Readiness probe failed: HTTP probe failed with statuscode: 503 11m Warning ProbeError pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj Readiness probe error: HTTP probe failed with statuscode: 503... 10m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 10m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 8m39s Normal SuccessfulCreate replicaset/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7 Created pod: e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs 8m39s Normal ScalingReplicaSet deployment/e2e-worker-single-main-ddcab1-openshift-default Scaled up replica set e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7 from 0 to 1 8m39s Normal AddedLabel gateway/e2e-worker-single-main-ddcab1 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-ddcab1 8m38s Normal AddedInterface pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Add eth0 [10.133.1.16/23] from ovn-kubernetes 8m34s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcb9b7 Error creating: pods "payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcb9b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1: serviceaccount "payload-processing-e2e-worker-single-main-ddcab1" not found 8m34s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcb9b7 Created pod: payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 8m34s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758 Error creating: pods "payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1: serviceaccount "payload-processing-e2e-worker-single-main-ddcab1" not found 8m34s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-ddcab1 Scaled up replica set payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcb9b7 from 0 to 1 8m34s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-ddcab1 Scaled up replica set payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758 from 0 to 1 8m33s Normal Created pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Container created 8m33s Normal Started pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Container started 8m33s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Add eth0 [10.133.1.18/23] from ovn-kubernetes 8m33s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 8m33s Normal Started pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Container started 8m33s Normal Created pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Container created 8m33s Normal Pulled pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 8m33s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Add eth0 [10.133.1.19/23] from ovn-kubernetes 8m33s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758 Created pod: payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph 8m22s Normal Pulled pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m22s Normal Started pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Container started 8m22s Warning ProbeError pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Readiness probe error: Get "http://10.133.1.16:15021/healthz/ready": dial tcp 10.133.1.16:15021: connect: connection refused... 8m22s Warning Unhealthy pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Readiness probe failed: Get "http://10.133.1.16:15021/healthz/ready": dial tcp 10.133.1.16:15021: connect: connection refused 8m22s Normal Created pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Container created 8m16s Warning ProbeError pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Startup probe error: Get "http://10.133.1.16:15021/healthz/ready": dial tcp 10.133.1.16:15021: connect: connection refused... 8m16s Warning Unhealthy pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Startup probe failed: Get "http://10.133.1.16:15021/healthz/ready": dial tcp 10.133.1.16:15021: connect: connection refused 8m9s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ddcab1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m9s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ddcab1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m39s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ddcab1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 7m39s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ddcab1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 7m26s Normal Killing pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Stopping container payload-processing 7m26s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Stopping container payload-pre-processing 7m17s Normal Killing pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Stopping container istio-proxy 7m15s Warning ProbeError pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Readiness probe error: HTTP probe failed with statuscode: 503... 7m15s Warning Unhealthy pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs Readiness probe failed: HTTP probe failed with statuscode: 503 7m11s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m11s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m7s Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-4bknm Readiness probe failed: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m7s Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-4bknm Readiness probe error: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m31s Normal AddedLabel gateway/e2e-ait-069c47a6 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-069c47a6 5m31s Normal ScalingReplicaSet deployment/e2e-ait-069c47a6-openshift-default Scaled up replica set e2e-ait-069c47a6-openshift-default-f9dcdf969 from 0 to 1 5m31s Normal SuccessfulCreate replicaset/e2e-ait-069c47a6-openshift-default-f9dcdf969 Created pod: e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl 5m31s Normal AddedInterface pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Add eth0 [10.133.1.29/23] from ovn-kubernetes 5m25s Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-069c47a6-96db766f8 Created pod: payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd 5m25s Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59 Error creating: pods "payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-069c47a6: serviceaccount "payload-processing-e2e-ait-069c47a6" not found 5m25s Warning FailedCreate replicaset/payload-processing-e2e-ait-069c47a6-96db766f8 Error creating: pods "payload-processing-e2e-ait-069c47a6-96db766f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-069c47a6: serviceaccount "payload-processing-e2e-ait-069c47a6" not found 5m25s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-069c47a6 Scaled up replica set payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59 from 0 to 1 5m25s Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-069c47a6 Scaled up replica set payload-processing-e2e-ait-069c47a6-96db766f8 from 0 to 1 5m25s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59 Created pod: payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 5m24s Normal Started pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Container started 5m24s Normal Created pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Container created 5m24s Normal AddedInterface pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Add eth0 [10.133.1.32/23] from ovn-kubernetes 5m24s Normal Created pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Container created 5m24s Normal Pulled pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m24s Normal Started pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Container started 5m24s Normal AddedInterface pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Add eth0 [10.133.1.31/23] from ovn-kubernetes 5m24s Normal Pulled pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m13s Warning Unhealthy pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Readiness probe failed: Get "http://10.133.1.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m13s Warning ProbeError pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Readiness probe error: Get "http://10.133.1.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m12s Normal Started pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Container started 5m12s Normal Pulled pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m12s Normal Created pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Container created 5m6s Warning ProbeError pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Startup probe error: Get "http://10.133.1.29:15021/healthz/ready": dial tcp 10.133.1.29:15021: connect: connection refused... 5m6s Warning Unhealthy pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Startup probe failed: Get "http://10.133.1.29:15021/healthz/ready": dial tcp 10.133.1.29:15021: connect: connection refused 5m2s Normal Killing pod/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8 Stopping container payload-pre-processing 5m2s Normal Killing pod/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd Stopping container payload-processing 5m1s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-069c47a6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m1s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-069c47a6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m56s Normal Killing pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl Stopping container istio-proxy 4m54s Normal AddedInterface pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Add eth0 [10.133.1.33/23] from ovn-kubernetes 4m54s Normal SuccessfulCreate replicaset/e2e-authcfg-http-783444-openshift-default-545d7f66fc Created pod: e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph 4m54s Normal ScalingReplicaSet deployment/e2e-authcfg-http-783444-openshift-default Scaled up replica set e2e-authcfg-http-783444-openshift-default-545d7f66fc from 0 to 1 4m54s Normal AddedLabel gateway/e2e-authcfg-http-783444 Added label istio.io/rev=openshift-gateway to gateway e2e-authcfg-http-783444 4m49s Normal ScalingReplicaSet deployment/payload-processing-e2e-authcfg-http-783444 Scaled up replica set payload-processing-e2e-authcfg-http-783444-69d6d6f468 from 0 to 1 4m48s Normal Started pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Container started 4m48s Normal AddedInterface pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Add eth0 [10.133.1.35/23] from ovn-kubernetes 4m48s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-authcfg-http-783444 Scaled up replica set payload-pre-processing-e2e-authcfg-http-783444-5f746687 from 0 to 1 4m48s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-authcfg-http-783444-5f746687 Created pod: payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf 4m48s Warning FailedCreate replicaset/payload-pre-processing-e2e-authcfg-http-783444-5f746687 Error creating: pods "payload-pre-processing-e2e-authcfg-http-783444-5f746687-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-783444: serviceaccount "payload-processing-e2e-authcfg-http-783444" not found 4m48s Normal Created pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Container created 4m48s Warning FailedCreate replicaset/payload-processing-e2e-authcfg-http-783444-69d6d6f468 Error creating: pods "payload-processing-e2e-authcfg-http-783444-69d6d6f468-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-783444: serviceaccount "payload-processing-e2e-authcfg-http-783444" not found 4m48s Normal SuccessfulCreate replicaset/payload-processing-e2e-authcfg-http-783444-69d6d6f468 Created pod: payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg 4m48s Normal AddedInterface pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Add eth0 [10.133.1.36/23] from ovn-kubernetes 4m48s Normal Pulled pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 4m48s Normal Pulled pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 4m47s Normal Started pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Container started 4m47s Normal Created pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Container created 4m37s Warning Unhealthy pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Readiness probe failed: Get "http://10.133.1.33:15021/healthz/ready": dial tcp 10.133.1.33:15021: connect: connection refused 4m37s Warning ProbeError pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Readiness probe error: Get "http://10.133.1.33:15021/healthz/ready": dial tcp 10.133.1.33:15021: connect: connection refused... 4m36s Normal Pulled pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m36s Normal Started pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Container started 4m36s Normal Created pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Container created 4m31s Warning Unhealthy pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Startup probe failed: Get "http://10.133.1.33:15021/healthz/ready": dial tcp 10.133.1.33:15021: connect: connection refused 4m31s Warning ProbeError pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Startup probe error: Get "http://10.133.1.33:15021/healthz/ready": dial tcp 10.133.1.33:15021: connect: connection refused... 4m24s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-783444-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m24s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-783444-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m9s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-783444-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m9s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-783444-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 3m4s Normal Killing pod/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg Stopping container payload-processing 3m4s Normal Killing pod/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf Stopping container payload-pre-processing 2m49s Normal Killing pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Stopping container istio-proxy 2m45s Warning ProbeError pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Readiness probe error: HTTP probe failed with statuscode: 503... 2m45s Warning Unhealthy pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph Readiness probe failed: HTTP probe failed with statuscode: 503 2m42s Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Readiness probe failed: Get "http://10.134.0.159:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m42s Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-h2w59 Readiness probe error: Get "http://10.134.0.159:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m42s Warning Unhealthy pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Readiness probe failed: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m42s Warning ProbeError pod/maas-default-gateway-openshift-default-77f556d647-f5zxr Readiness probe error: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...