=== events in openshift-ingress (2026-10-30T20:24:14Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 62m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k to ip-10-0-135-68.ec2.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 to ip-10-0-135-68.ec2.internal 28m Normal Scheduled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Successfully assigned openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 to ip-10-0-130-217.ec2.internal 54m Normal Scheduled pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Successfully assigned openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 to ip-10-0-135-68.ec2.internal 28m Normal Scheduled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Successfully assigned openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt to ip-10-0-130-217.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr to ip-10-0-140-235.ec2.internal 28m Normal Scheduled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Successfully assigned openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 to ip-10-0-130-217.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx to ip-10-0-130-217.ec2.internal 55m Normal Scheduled pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Successfully assigned openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc to ip-10-0-130-217.ec2.internal 59m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 to ip-10-0-135-68.ec2.internal 52m Normal Scheduled pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Successfully assigned openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 to ip-10-0-135-68.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 to ip-10-0-135-68.ec2.internal 31m Normal Scheduled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Successfully assigned openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml to ip-10-0-135-68.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb to ip-10-0-135-68.ec2.internal 60m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws to ip-10-0-135-68.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 to ip-10-0-140-235.ec2.internal 58m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt to ip-10-0-140-235.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk to ip-10-0-130-217.ec2.internal 51m Normal Scheduled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Successfully assigned openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk to ip-10-0-130-217.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Successfully assigned openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj to ip-10-0-140-235.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Successfully assigned openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j to ip-10-0-135-68.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf to ip-10-0-135-68.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd to ip-10-0-130-217.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p to ip-10-0-140-235.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r to ip-10-0-140-235.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Successfully assigned openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 to ip-10-0-130-217.ec2.internal 9m50s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh to ip-10-0-135-68.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz to ip-10-0-140-235.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Successfully assigned openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g to ip-10-0-140-235.ec2.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb to ip-10-0-135-68.ec2.internal 60m Normal Scheduled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Successfully assigned openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Successfully assigned openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f to ip-10-0-135-68.ec2.internal 58m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp to ip-10-0-130-217.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Successfully assigned openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s to ip-10-0-135-68.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Successfully assigned openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt to ip-10-0-130-217.ec2.internal 33m Normal Scheduled pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Successfully assigned openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz to ip-10-0-135-68.ec2.internal 53m Normal Scheduled pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Successfully assigned openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx to ip-10-0-140-235.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r to ip-10-0-140-235.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths to ip-10-0-135-68.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 to ip-10-0-140-235.ec2.internal 62m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct to ip-10-0-135-68.ec2.internal 63m Normal Scheduled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Successfully assigned openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt to ip-10-0-140-235.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh to ip-10-0-135-68.ec2.internal 53m Normal Scheduled pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Successfully assigned openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 to ip-10-0-130-217.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m to ip-10-0-135-68.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn to ip-10-0-130-217.ec2.internal 51m Normal Scheduled pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Successfully assigned openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns to ip-10-0-140-235.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w to ip-10-0-140-235.ec2.internal 61m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 to ip-10-0-130-217.ec2.internal 57m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 to ip-10-0-140-235.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr to ip-10-0-135-68.ec2.internal 52m Normal Scheduled pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Successfully assigned openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf to ip-10-0-140-235.ec2.internal 32m Normal Scheduled pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Successfully assigned openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf to ip-10-0-135-68.ec2.internal 41m Normal Scheduled pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Successfully assigned openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk to ip-10-0-135-68.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 to ip-10-0-140-235.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd to ip-10-0-135-68.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx to ip-10-0-130-217.ec2.internal 19m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb to ip-10-0-140-235.ec2.internal 63m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd to ip-10-0-140-235.ec2.internal 48m Normal Scheduled pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Successfully assigned openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd to ip-10-0-140-235.ec2.internal 58m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 to ip-10-0-130-217.ec2.internal 64m Normal Scheduled pod/payload-processing-5f78687d5b-rspbb Successfully assigned openshift-ingress/payload-processing-5f78687d5b-rspbb to ip-10-0-140-235.ec2.internal 36m Normal Scheduled pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Successfully assigned openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Successfully assigned openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl to ip-10-0-130-217.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 to ip-10-0-130-217.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67 to ip-10-0-130-217.ec2.internal 55m Normal Scheduled pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Successfully assigned openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb to ip-10-0-140-235.ec2.internal 57m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s to ip-10-0-130-217.ec2.internal 60m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph to ip-10-0-135-68.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62 to ip-10-0-135-68.ec2.internal 28m Normal Scheduled pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Successfully assigned openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 to ip-10-0-135-68.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 to ip-10-0-130-217.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Successfully assigned openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 to ip-10-0-140-235.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt to ip-10-0-140-235.ec2.internal 58m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm to ip-10-0-135-68.ec2.internal 58m Normal Scheduled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Successfully assigned openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg to ip-10-0-140-235.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk to ip-10-0-140-235.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk to ip-10-0-135-68.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz to ip-10-0-135-68.ec2.internal 80m Normal Scheduled pod/router-default-77d478f789-d787c Successfully assigned openshift-ingress/router-default-77d478f789-d787c to ip-10-0-140-235.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q to ip-10-0-140-235.ec2.internal 25m Normal Scheduled pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Successfully assigned openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 to ip-10-0-135-68.ec2.internal 32m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-frwdk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh to ip-10-0-135-68.ec2.internal 46m Normal Scheduled pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Successfully assigned openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw to ip-10-0-140-235.ec2.internal 80m Warning FailedScheduling pod/router-default-77d478f789-d787c no nodes available to schedule pods 86m Warning FailedScheduling pod/router-default-77d478f789-d787c no nodes available to schedule pods 50m Normal Scheduled pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 to ip-10-0-135-68.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k to ip-10-0-135-68.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv to ip-10-0-135-68.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 to ip-10-0-135-68.ec2.internal 32m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 to ip-10-0-135-68.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z to ip-10-0-140-235.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr to ip-10-0-135-68.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc to ip-10-0-135-68.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 to ip-10-0-135-68.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Successfully assigned openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t to ip-10-0-130-217.ec2.internal 57m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm to ip-10-0-135-68.ec2.internal 24m Normal Scheduled pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Successfully assigned openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 to ip-10-0-135-68.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 to ip-10-0-130-217.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz to ip-10-0-140-235.ec2.internal 32m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft to ip-10-0-135-68.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g to ip-10-0-140-235.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s to ip-10-0-135-68.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv to ip-10-0-140-235.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx to ip-10-0-130-217.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct to ip-10-0-130-217.ec2.internal 62m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k to ip-10-0-135-68.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Successfully assigned openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq to ip-10-0-140-235.ec2.internal 60m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg to ip-10-0-130-217.ec2.internal 61m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 to ip-10-0-135-68.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w to ip-10-0-130-217.ec2.internal 59m Normal Scheduled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Successfully assigned openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Successfully assigned openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz to ip-10-0-135-68.ec2.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj to ip-10-0-135-68.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Successfully assigned openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m to ip-10-0-135-68.ec2.internal 41m Normal Scheduled pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Successfully assigned openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp to ip-10-0-140-235.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj to ip-10-0-130-217.ec2.internal 55m Normal Scheduled pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 to ip-10-0-140-235.ec2.internal 19m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-czfvt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt to ip-10-0-135-68.ec2.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 to ip-10-0-130-217.ec2.internal 45m Normal Scheduled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Successfully assigned openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h to ip-10-0-140-235.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Successfully assigned openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld to ip-10-0-130-217.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs to ip-10-0-140-235.ec2.internal 51m Normal Scheduled pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 to ip-10-0-135-68.ec2.internal 31m Normal Scheduled pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Successfully assigned openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd to ip-10-0-140-235.ec2.internal 68m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn to ip-10-0-130-217.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4 to ip-10-0-140-235.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb to ip-10-0-130-217.ec2.internal 64m Normal Scheduled pod/payload-pre-processing-7b98bb678c-mptvz Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-mptvz to ip-10-0-140-235.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Successfully assigned openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt to ip-10-0-130-217.ec2.internal 35m Normal Scheduled pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Successfully assigned openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r to ip-10-0-140-235.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 to ip-10-0-140-235.ec2.internal 52m Normal Scheduled pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-8k46t Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t to ip-10-0-130-217.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Successfully assigned openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 to ip-10-0-130-217.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Successfully assigned openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 to ip-10-0-135-68.ec2.internal 43m Normal Scheduled pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Successfully assigned openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj to ip-10-0-140-235.ec2.internal 30m Normal Scheduled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Successfully assigned openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz to ip-10-0-140-235.ec2.internal 34m Normal Scheduled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Successfully assigned openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 to ip-10-0-135-68.ec2.internal 72m Normal Scheduled pod/kube-auth-proxy-6c7d47d6c9-jvqgk Successfully assigned openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Successfully assigned openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn to ip-10-0-140-235.ec2.internal 35m Normal Scheduled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Successfully assigned openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd to ip-10-0-130-217.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp to ip-10-0-135-68.ec2.internal 72m Normal Scheduled pod/kube-auth-proxy-6c7d47d6c9-dbd6g Successfully assigned openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g to ip-10-0-135-68.ec2.internal 48m Normal Scheduled pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Successfully assigned openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 to ip-10-0-135-68.ec2.internal 52m Normal Scheduled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Successfully assigned openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Successfully assigned openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf to ip-10-0-135-68.ec2.internal 53m Normal Scheduled pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Successfully assigned openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 to ip-10-0-130-217.ec2.internal 40m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 to ip-10-0-140-235.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs to ip-10-0-135-68.ec2.internal 42m Normal Scheduled pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Successfully assigned openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz to ip-10-0-140-235.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr to ip-10-0-140-235.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr to ip-10-0-140-235.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx to ip-10-0-135-68.ec2.internal 40m Normal Scheduled pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Successfully assigned openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd to ip-10-0-140-235.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-xljwk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk to ip-10-0-140-235.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr to ip-10-0-135-68.ec2.internal 9m50s Normal Scheduled pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm to ip-10-0-135-68.ec2.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg to ip-10-0-135-68.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w to ip-10-0-140-235.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 to ip-10-0-135-68.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 to ip-10-0-135-68.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 to ip-10-0-140-235.ec2.internal 61m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq to ip-10-0-140-235.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 to ip-10-0-135-68.ec2.internal 53m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x to ip-10-0-135-68.ec2.internal 45m Normal Scheduled pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Successfully assigned openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr to ip-10-0-140-235.ec2.internal 71m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Successfully assigned openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 to ip-10-0-130-217.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv to ip-10-0-130-217.ec2.internal 55m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm to ip-10-0-135-68.ec2.internal 51m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb to ip-10-0-130-217.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt to ip-10-0-140-235.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d to ip-10-0-140-235.ec2.internal 60m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv to ip-10-0-140-235.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Successfully assigned openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 to ip-10-0-135-68.ec2.internal 61m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh to ip-10-0-135-68.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26 to ip-10-0-140-235.ec2.internal 35m Normal Scheduled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Successfully assigned openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 to ip-10-0-140-235.ec2.internal 52m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td to ip-10-0-135-68.ec2.internal 52m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 to ip-10-0-140-235.ec2.internal 19m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5 to ip-10-0-135-68.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Successfully assigned openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv to ip-10-0-135-68.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-scx8v Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v to ip-10-0-135-68.ec2.internal 62m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz to ip-10-0-135-68.ec2.internal 60m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 to ip-10-0-130-217.ec2.internal 61m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w to ip-10-0-140-235.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d to ip-10-0-140-235.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h to ip-10-0-135-68.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx to ip-10-0-135-68.ec2.internal 63m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Successfully assigned openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc to ip-10-0-130-217.ec2.internal 45m Normal Scheduled pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Successfully assigned openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8 to ip-10-0-140-235.ec2.internal 32m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 to ip-10-0-130-217.ec2.internal 53m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4 to ip-10-0-130-217.ec2.internal 29m Normal Scheduled pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Successfully assigned openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl to ip-10-0-130-217.ec2.internal 52m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk to ip-10-0-140-235.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Successfully assigned openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf to ip-10-0-130-217.ec2.internal 34m Normal Scheduled pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Successfully assigned openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt to ip-10-0-140-235.ec2.internal 72m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c to ip-10-0-130-217.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc to ip-10-0-140-235.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb to ip-10-0-135-68.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd to ip-10-0-130-217.ec2.internal 53m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 to ip-10-0-135-68.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv to ip-10-0-140-235.ec2.internal 62m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r to ip-10-0-130-217.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 to ip-10-0-135-68.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj to ip-10-0-135-68.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd to ip-10-0-135-68.ec2.internal 61m Normal Scheduled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Successfully assigned openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm to ip-10-0-130-217.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z to ip-10-0-140-235.ec2.internal 51m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l to ip-10-0-130-217.ec2.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-nv972 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972 to ip-10-0-130-217.ec2.internal 60m Normal Scheduled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Successfully assigned openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c to ip-10-0-140-235.ec2.internal 10m Normal Scheduled pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Successfully assigned openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t to ip-10-0-135-68.ec2.internal 49m Normal Scheduled pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Successfully assigned openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h to ip-10-0-140-235.ec2.internal 51m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst to ip-10-0-135-68.ec2.internal 14m Normal Scheduled pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Successfully assigned openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs to ip-10-0-135-68.ec2.internal 40m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc to ip-10-0-130-217.ec2.internal 52m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 to ip-10-0-135-68.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Successfully assigned openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l to ip-10-0-135-68.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj to ip-10-0-135-68.ec2.internal 33m Normal Scheduled pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Successfully assigned openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s to ip-10-0-135-68.ec2.internal 47m Normal Scheduled pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Successfully assigned openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw to ip-10-0-140-235.ec2.internal 59m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z to ip-10-0-130-217.ec2.internal 52m Normal Scheduled pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Successfully assigned openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt to ip-10-0-135-68.ec2.internal 45m Normal Scheduled pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Successfully assigned openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j to ip-10-0-135-68.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Successfully assigned openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx to ip-10-0-140-235.ec2.internal 56m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 to ip-10-0-135-68.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v to ip-10-0-130-217.ec2.internal 30m Normal Scheduled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Successfully assigned openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx to ip-10-0-135-68.ec2.internal 61m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Successfully assigned openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j to ip-10-0-130-217.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j to ip-10-0-140-235.ec2.internal 34m Normal Scheduled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Successfully assigned openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 to ip-10-0-140-235.ec2.internal 39m Normal Scheduled pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Successfully assigned openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t to ip-10-0-140-235.ec2.internal 63m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz to ip-10-0-135-68.ec2.internal 54m Normal Scheduled pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Successfully assigned openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb to ip-10-0-140-235.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf to ip-10-0-140-235.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 to ip-10-0-140-235.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc to ip-10-0-140-235.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l to ip-10-0-140-235.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj to ip-10-0-135-68.ec2.internal 46m Normal Scheduled pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Successfully assigned openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k to ip-10-0-135-68.ec2.internal 43m Normal Scheduled pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Successfully assigned openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg to ip-10-0-140-235.ec2.internal 54m Normal Scheduled pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Successfully assigned openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb to ip-10-0-140-235.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj to ip-10-0-135-68.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l to ip-10-0-140-235.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff to ip-10-0-140-235.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Successfully assigned openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 to ip-10-0-140-235.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Successfully assigned openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m to ip-10-0-130-217.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n to ip-10-0-140-235.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp to ip-10-0-135-68.ec2.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v to ip-10-0-135-68.ec2.internal 33m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx to ip-10-0-140-235.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Successfully assigned openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv to ip-10-0-135-68.ec2.internal 32m Normal Scheduled pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs to ip-10-0-135-68.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8 to ip-10-0-135-68.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 to ip-10-0-135-68.ec2.internal 33m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 to ip-10-0-135-68.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 to ip-10-0-130-217.ec2.internal 50m Normal Scheduled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Successfully assigned openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt to ip-10-0-140-235.ec2.internal 32m Normal Scheduled pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 to ip-10-0-135-68.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 to ip-10-0-135-68.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm to ip-10-0-140-235.ec2.internal 72m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-5rm8g Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g to ip-10-0-140-235.ec2.internal 44m Normal Scheduled pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Successfully assigned openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m to ip-10-0-130-217.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff to ip-10-0-140-235.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z to ip-10-0-135-68.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz to ip-10-0-140-235.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 to ip-10-0-135-68.ec2.internal 86m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-77d478f789 from 0 to 1 86m Normal SuccessfulCreate replicaset/router-default-77d478f789 Created pod: router-default-77d478f789-d787c 86m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 86m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 86m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 80m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 80m Warning FailedMount pod/router-default-77d478f789-d787c MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 80m Warning FailedMount pod/router-default-77d478f789-d787c MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 80m Warning FailedMount pod/router-default-77d478f789-d787c MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 80m Warning FailedMount pod/router-default-77d478f789-d787c MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 80m Normal Created pod/router-default-77d478f789-d787c Container created 80m Normal Started pod/router-default-77d478f789-d787c Container started 80m Normal Pulled pod/router-default-77d478f789-d787c Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 80m Normal AddedInterface pod/router-default-77d478f789-d787c Add eth0 [10.132.0.13/23] from ovn-kubernetes 72m Normal SuccessfulCreate replicaset/kube-auth-proxy-6c7d47d6c9 Created pod: kube-auth-proxy-6c7d47d6c9-dbd6g 72m Normal AddedInterface pod/kube-auth-proxy-6c7d47d6c9-jvqgk Add eth0 [10.132.0.32/23] from ovn-kubernetes 72m Normal Pulling pod/kube-auth-proxy-6c7d47d6c9-dbd6g Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 72m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-6c7d47d6c9 from 0 to 2 72m Normal AddedInterface pod/kube-auth-proxy-6c7d47d6c9-dbd6g Add eth0 [10.134.0.21/23] from ovn-kubernetes 72m Normal Pulling pod/kube-auth-proxy-6c7d47d6c9-jvqgk Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 72m Normal SuccessfulCreate replicaset/kube-auth-proxy-6c7d47d6c9 Created pod: kube-auth-proxy-6c7d47d6c9-jvqgk 72m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 72m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-5rm8g 72m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-5rm8g Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 72m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-5rm8g Add eth0 [10.132.0.33/23] from ovn-kubernetes 72m Normal Started pod/kube-auth-proxy-6c7d47d6c9-dbd6g Container started 72m Normal Created pod/kube-auth-proxy-6c7d47d6c9-jvqgk Container created 72m Normal Started pod/kube-auth-proxy-6c7d47d6c9-jvqgk Container started 72m Normal Pulled pod/kube-auth-proxy-6c7d47d6c9-jvqgk Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.449s (2.45s including waiting). Image size: 188904677 bytes. 72m Normal Created pod/kube-auth-proxy-6c7d47d6c9-dbd6g Container created 72m Normal Pulled pod/kube-auth-proxy-6c7d47d6c9-dbd6g Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.374s (2.374s including waiting). Image size: 188904677 bytes. 72m Normal Started pod/istiod-openshift-gateway-7876bfcb74-5rm8g Container started 72m Normal Created pod/istiod-openshift-gateway-7876bfcb74-5rm8g Container created 72m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-5rm8g Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.604s (2.604s including waiting). Image size: 185476477 bytes. 72m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-7556cf8c57 from 0 to 1 72m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Add eth0 [10.133.0.28/23] from ovn-kubernetes 72m Normal Pulling pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 72m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-5rm8g Readiness probe error: HTTP probe failed with statuscode: 503... 72m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-7556cf8c57 Created pod: data-science-gateway-data-science-gateway-class-7556cf8c574v98c 72m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-5rm8g Readiness probe failed: HTTP probe failed with statuscode: 503 72m Normal Pulled pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.375s (3.375s including waiting). Image size: 183555333 bytes. 72m Normal Created pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Container created 72m Normal Started pod/data-science-gateway-data-science-gateway-class-7556cf8c574v98c Container started 72m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 72m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 71m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 71m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 71m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 71m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-kzmjv 71m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Container created 71m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Container started 71m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 71m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 71m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Add eth0 [10.132.0.34/23] from ovn-kubernetes 71m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 71m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Readiness probe error: HTTP probe failed with statuscode: 503... 71m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv Readiness probe failed: HTTP probe failed with statuscode: 503 71m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 71m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-5rm8g 71m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-5rm8g Stopping container discovery 71m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 71m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 68m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-cnvpn 68m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 68m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Add eth0 [10.133.0.35/23] from ovn-kubernetes 68m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 0 to 1 66m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 66m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Container started 66m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Container created 66m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Startup probe failed: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused 66m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Startup probe error: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused... 66m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 66m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-x4r26 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-r9r2z 64m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 1 to 4 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-nlrv8 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Add eth0 [10.132.0.48/23] from ovn-kubernetes 64m Normal Pulling pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Add eth0 [10.134.0.38/23] from ovn-kubernetes 64m Normal Pulling pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 64m Normal Pulling pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Add eth0 [10.132.0.49/23] from ovn-kubernetes 64m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Container created 64m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.599s (2.599s including waiting). Image size: 183555333 bytes. 64m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.993s (2.993s including waiting). Image size: 183555333 bytes. 64m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.9s (2.9s including waiting). Image size: 183555333 bytes. 64m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Container started 64m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Container started 64m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Container created 64m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Container created 64m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Container started 64m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 64m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 64m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused... 64m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused 64m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused... 64m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused 64m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 4 to 6 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-g2h62 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-j8c67 64m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5f78687d5b from 0 to 1 64m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Add eth0 [10.134.0.40/23] from ovn-kubernetes 64m Normal AddedInterface pod/payload-processing-5f78687d5b-rspbb Add eth0 [10.132.0.52/23] from ovn-kubernetes 64m Normal Pulling pod/payload-pre-processing-7b98bb678c-mptvz Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 64m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-mptvz Add eth0 [10.132.0.51/23] from ovn-kubernetes 64m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Container created 64m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Container started 64m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-mptvz 64m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 64m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Container created 64m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 64m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Container started 64m Normal Pulling pod/payload-processing-5f78687d5b-rspbb Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 64m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 64m Warning FailedCreate replicaset/payload-processing-5f78687d5b Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Add eth0 [10.133.0.39/23] from ovn-kubernetes 64m Normal SuccessfulCreate replicaset/payload-processing-5f78687d5b Created pod: payload-processing-5f78687d5b-rspbb 64m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 64m Normal Created pod/payload-processing-5f78687d5b-rspbb Container created 64m Normal Created pod/payload-pre-processing-7b98bb678c-mptvz Container created 64m Normal Pulled pod/payload-pre-processing-7b98bb678c-mptvz Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.157s (3.157s including waiting). Image size: 125353965 bytes. 64m Normal Started pod/payload-processing-5f78687d5b-rspbb Container started 64m Normal Started pod/payload-pre-processing-7b98bb678c-mptvz Container started 64m Normal Pulled pod/payload-processing-5f78687d5b-rspbb Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.072s (3.072s including waiting). Image size: 125353965 bytes. 64m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused 64m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused 64m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused... 64m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused... 63m Normal AddedLabel gateway/e2e-worker-w2-9efc72 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-9efc72 63m Normal ScalingReplicaSet deployment/e2e-worker-w3-249dd4-openshift-default Scaled up replica set e2e-worker-w3-249dd4-openshift-default-59f7849b69 from 0 to 1 63m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 0 to 1 63m Normal AddedLabel gateway/e2e-worker-w1-5ac614 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-5ac614 63m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz 63m Normal AddedLabel gateway/e2e-worker-w5-e67b84 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-e67b84 63m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 0 to 1 63m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd 63m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 0 to 1 63m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 63m Normal AddedLabel gateway/e2e-worker-w3-249dd4 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-249dd4 63m Normal AddedLabel gateway/e2e-worker-w6-6e0c55 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-6e0c55 63m Normal SuccessfulCreate replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Created pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt 63m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Add eth0 [10.134.0.41/23] from ovn-kubernetes 63m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Add eth0 [10.132.0.56/23] from ovn-kubernetes 63m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 0 to 1 63m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Add eth0 [10.132.0.54/23] from ovn-kubernetes 63m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Add eth0 [10.134.0.42/23] from ovn-kubernetes 63m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j 63m Normal AddedInterface pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Add eth0 [10.132.0.55/23] from ovn-kubernetes 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Add eth0 [10.132.0.57/23] from ovn-kubernetes 62m Normal AddedInterface pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Add eth0 [10.134.0.43/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-9efc72-79b9b67494 Created pod: payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f Error creating: pods "payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-9efc72: serviceaccount "payload-processing-e2e-worker-w2-9efc72" not found 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f Created pod: payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-9efc72 Scaled up replica set payload-pre-processing-e2e-worker-w2-9efc72-899979b9f from 0 to 1 62m Normal Pulling pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-9efc72 Scaled up replica set payload-processing-e2e-worker-w2-9efc72-79b9b67494 from 0 to 1 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-9efc72-79b9b67494 Error creating: pods "payload-processing-e2e-worker-w2-9efc72-79b9b67494-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-9efc72: serviceaccount "payload-processing-e2e-worker-w2-9efc72" not found 62m Normal Started pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Container started 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-6e0c55 Scaled up replica set payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb from 0 to 1 62m Normal Created pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Container created 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-6e0c55 Scaled up replica set payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947 from 0 to 1 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb Error creating: pods "payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-6e0c55: serviceaccount "payload-processing-e2e-worker-w6-6e0c55" not found 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb Created pod: payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-5ac614 Scaled up replica set payload-processing-e2e-worker-w1-5ac614-67c554ff88 from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-5ac614 Scaled up replica set payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5 from 0 to 1 62m Normal Pulling pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947 Created pod: payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947 Error creating: pods "payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-6e0c55: serviceaccount "payload-processing-e2e-worker-w6-6e0c55" not found 62m Normal Pulling pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Add eth0 [10.134.0.45/23] from ovn-kubernetes 62m Normal AddedInterface pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Add eth0 [10.134.0.44/23] from ovn-kubernetes 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-5ac614-67c554ff88 Error creating: pods "payload-processing-e2e-worker-w1-5ac614-67c554ff88-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-5ac614: serviceaccount "payload-processing-e2e-worker-w1-5ac614" not found 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5 Error creating: pods "payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-5ac614: serviceaccount "payload-processing-e2e-worker-w1-5ac614" not found 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5 Created pod: payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 543ms (543ms including waiting). Image size: 125353965 bytes. 62m Normal Started pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Container started 62m Normal Pulled pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 650ms (650ms including waiting). Image size: 125353965 bytes. 62m Normal Created pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Container created 62m Normal Started pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Container started 62m Normal Created pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Container created 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Add eth0 [10.134.0.46/23] from ovn-kubernetes 62m Normal Started pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Container started 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-5ac614-67c554ff88 Created pod: payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 62m Normal Created pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Container created 62m Normal Started pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Container started 62m Normal Created pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Container created 62m Normal Pulled pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Add eth0 [10.132.0.59/23] from ovn-kubernetes 62m Normal Pulled pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.304s (2.305s including waiting). Image size: 125353965 bytes. 62m Normal Started pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Container started 62m Normal Created pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Container created 62m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb 62m Normal AddedLabel gateway/e2e-worker-w4-b97379 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-b97379 62m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d 62m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled up replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 0 to 1 62m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled up replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 0 to 1 62m Normal AddedLabel gateway/e2e-worker-w0-c5449c Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-c5449c 62m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Add eth0 [10.134.0.47/23] from ovn-kubernetes 62m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Add eth0 [10.132.0.60/23] from ovn-kubernetes 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff Error creating: pods "payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-e67b84: serviceaccount "payload-processing-e2e-worker-w5-e67b84" not found 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff Created pod: payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-e67b84-9578b57dd Error creating: pods "payload-processing-e2e-worker-w5-e67b84-9578b57dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-e67b84: serviceaccount "payload-processing-e2e-worker-w5-e67b84" not found 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-e67b84-9578b57dd Created pod: payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-e67b84 Scaled up replica set payload-processing-e2e-worker-w5-e67b84-9578b57dd from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-e67b84 Scaled up replica set payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff from 0 to 1 62m Normal Pulling pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 62m Normal AddedInterface pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Add eth0 [10.133.0.40/23] from ovn-kubernetes 62m Normal Started pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Container started 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Add eth0 [10.134.0.48/23] from ovn-kubernetes 62m Normal Created pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Container created 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c Created pod: payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z 62m Normal Started pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Container started 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-b97379-58647c55b4 Error creating: pods "payload-processing-e2e-worker-w4-b97379-58647c55b4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-b97379: serviceaccount "payload-processing-e2e-worker-w4-b97379" not found 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-b97379-58647c55b4 Created pod: payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-b97379 Scaled up replica set payload-processing-e2e-worker-w4-b97379-58647c55b4 from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-b97379 Scaled up replica set payload-pre-processing-e2e-worker-w4-b97379-95fdb8c from 0 to 1 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c Error creating: pods "payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-b97379: serviceaccount "payload-processing-e2e-worker-w4-b97379" not found 62m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Readiness probe error: Get "http://10.132.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 62m Normal Pulled pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.864s (1.864s including waiting). Image size: 125353965 bytes. 62m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Readiness probe failed: Get "http://10.132.0.54:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 62m Normal Created pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Container created 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87 Created pod: payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 62m Normal Created pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Container created 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-249dd4 Scaled up replica set payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87 from 0 to 1 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-249dd4 Scaled up replica set payload-processing-e2e-worker-w3-249dd4-548548cc5d from 0 to 1 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87 Error creating: pods "payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-249dd4: serviceaccount "payload-processing-e2e-worker-w3-249dd4" not found 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Add eth0 [10.134.0.49/23] from ovn-kubernetes 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-249dd4-548548cc5d Created pod: payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 62m Normal Started pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Container started 62m Normal Created pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Container created 62m Normal AddedInterface pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Add eth0 [10.133.0.41/23] from ovn-kubernetes 62m Normal Pulled pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-249dd4-548548cc5d Error creating: pods "payload-processing-e2e-worker-w3-249dd4-548548cc5d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-249dd4: serviceaccount "payload-processing-e2e-worker-w3-249dd4" not found 62m Normal Started pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Container started 62m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5 Created pod: payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx 62m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6 Error creating: pods "payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-c5449c: serviceaccount "payload-processing-e2e-worker-w0-c5449c" not found 62m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Container created 62m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Container started 62m Normal AddedInterface pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Add eth0 [10.133.0.43/23] from ovn-kubernetes 62m Normal Started pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Container started 62m Normal Created pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Container created 62m Normal Started pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Container started 62m Normal Created pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Container created 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Add eth0 [10.133.0.42/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-c5449c Scaled up replica set payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5 from 0 to 1 62m Normal Pulled pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6 Created pod: payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr 62m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-c5449c Scaled up replica set payload-processing-e2e-worker-w0-c5449c-7d46ff99d6 from 0 to 1 62m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5 Error creating: pods "payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-c5449c: serviceaccount "payload-processing-e2e-worker-w0-c5449c" not found 62m Normal Created pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Container created 62m Normal Started pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Container started 62m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal Started pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Container started 62m Normal Created pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Container created 62m Normal Pulled pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Add eth0 [10.134.0.50/23] from ovn-kubernetes 62m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Add eth0 [10.133.0.44/23] from ovn-kubernetes 62m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Startup probe failed: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Startup probe error: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused... 62m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Container started 62m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Container created 62m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Container started 62m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Container created 62m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused... 62m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Startup probe failed: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Startup probe error: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused... 62m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Container created 62m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Container started 62m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled up replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 1 to 4 62m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k 62m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r 62m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 62m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz 62m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Add eth0 [10.134.0.53/23] from ovn-kubernetes 62m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Container created 62m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Container started 62m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Container started 62m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Container created 62m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Container created 62m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Container started 62m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Add eth0 [10.133.0.48/23] from ovn-kubernetes 62m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Add eth0 [10.134.0.52/23] from ovn-kubernetes 62m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Readiness probe error: Get "http://10.132.0.60:15021/healthz/ready": read tcp 10.132.0.2:36160->10.132.0.60:15021: read: connection reset by peer... 62m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Readiness probe failed: Get "http://10.132.0.60:15021/healthz/ready": read tcp 10.132.0.2:36160->10.132.0.60:15021: read: connection reset by peer 62m Normal Created pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Container created 62m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Started pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Container started 62m Normal Pulled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Container started 62m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Container created 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 62m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Startup probe failed: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused 62m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Startup probe error: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused... 62m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Startup probe error: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused... 62m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused... 62m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Startup probe failed: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused 62m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Container started 62m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Container created 62m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Warning Unhealthy pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Startup probe error: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused... 62m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Startup probe failed: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused 62m Warning ProbeError pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused... 62m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Startup probe error: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused... 62m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Startup probe failed: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 61m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 61m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 1 to 2 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Container started 61m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Container created 61m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Add eth0 [10.133.0.52/23] from ovn-kubernetes 61m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused 61m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq 61m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-db77w 61m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm 61m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 8; reason: 61m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled up replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 4 to 8 61m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Container started 61m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Container created 61m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Add eth0 [10.134.0.58/23] from ovn-kubernetes 61m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Container created 61m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Container started 61m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Add eth0 [10.132.0.68/23] from ovn-kubernetes 61m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Add eth0 [10.132.0.67/23] from ovn-kubernetes 61m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Container created 61m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Container started 61m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Container started 61m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh 61m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Container created 61m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Add eth0 [10.133.0.53/23] from ovn-kubernetes 61m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Startup probe error: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused... 61m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 61m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Startup probe failed: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 61m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Add eth0 [10.132.0.69/23] from ovn-kubernetes 61m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 1 to 2 61m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj 61m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 61m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 1 to 2 61m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 61m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq 61m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Add eth0 [10.134.0.61/23] from ovn-kubernetes 61m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Container created 61m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Container started 61m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Container created 61m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Container started 61m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 61m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 61m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 61m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 61m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Add eth0 [10.132.0.70/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c 61m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Container started 61m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 61m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 1 to 2 61m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Container created 60m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 60m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Container started 60m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Container created 60m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Add eth0 [10.133.0.58/23] from ovn-kubernetes 60m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 60m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal AddedInterface pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Add eth0 [10.132.0.71/23] from ovn-kubernetes 60m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 (combined from similar events): Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv 60m Normal Pulled pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal SuccessfulCreate replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Created pod: e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 60m Normal Created pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Container created 60m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled up replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 8 to 10 60m Normal Started pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Container started 60m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph 60m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 2 to 3 60m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 60m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Container created 60m Normal Created pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Container created 60m Normal Pulled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Container started 60m Normal SuccessfulCreate replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Created pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np 60m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Add eth0 [10.134.0.64/23] from ovn-kubernetes 60m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 60m Normal ScalingReplicaSet deployment/e2e-worker-w3-249dd4-openshift-default Scaled up replica set e2e-worker-w3-249dd4-openshift-default-59f7849b69 from 1 to 2 60m Normal AddedInterface pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Add eth0 [10.134.0.65/23] from ovn-kubernetes 60m Normal Started pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Container started 60m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 60m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 60m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused... 60m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Startup probe error: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 60m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Startup probe failed: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 60m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled up replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 1 to 2 60m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 60m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 60m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Container started 60m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Container created 60m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Add eth0 [10.132.0.72/23] from ovn-kubernetes 60m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 60m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 60m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 60m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 2 to 3 60m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg 60m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Container created 60m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Add eth0 [10.133.0.61/23] from ovn-kubernetes 60m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Container started 60m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 60m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 59m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 6 to 10 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 (combined from similar events): Created pod: maas-default-gateway-openshift-default-6f54779779-frwdk 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-8k46t 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-scx8v 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-s2nd8 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-8k46t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-8k46t Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-scx8v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-scx8v Add eth0 [10.134.0.71/23] from ovn-kubernetes 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-frwdk Add eth0 [10.132.0.74/23] from ovn-kubernetes 59m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-8k46t Container started 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Add eth0 [10.132.0.73/23] from ovn-kubernetes 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-8k46t Add eth0 [10.133.0.64/23] from ovn-kubernetes 59m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-scx8v Container started 59m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-scx8v Container created 59m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Container created 59m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Container started 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-frwdk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-frwdk Container created 59m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-frwdk Container started 59m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Container created 59m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd 59m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 59m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled up replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 2 to 4 59m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Container started 59m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Add eth0 [10.132.0.75/23] from ovn-kubernetes 59m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Add eth0 [10.133.0.66/23] from ovn-kubernetes 59m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Container created 59m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Container started 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-frwdk Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-frwdk Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-scx8v Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-scx8v Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-8k46t Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-8k46t Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 59m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 59m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Add eth0 [10.134.0.74/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn 59m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Add eth0 [10.133.0.67/23] from ovn-kubernetes 59m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 2 to 4 59m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Container started 59m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Container created 59m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Container created 59m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Container started 59m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 59m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 59m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 59m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 59m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 59m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 2 to 3 59m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Add eth0 [10.132.0.76/23] from ovn-kubernetes 59m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 3 to 5 59m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 59m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Add eth0 [10.134.0.76/23] from ovn-kubernetes 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 59m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 59m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj 59m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Container created 59m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Container started 59m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Container created 59m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Container started 59m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 3 to 5 59m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 59m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 59m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Container created 59m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Container started 59m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Add eth0 [10.132.0.77/23] from ovn-kubernetes 59m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Add eth0 [10.134.0.77/23] from ovn-kubernetes 59m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Add eth0 [10.133.0.68/23] from ovn-kubernetes 59m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Container started 59m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Container started 59m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Container created 59m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Container created 59m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Startup probe error: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Startup probe failed: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Startup probe error: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Startup probe failed: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 59m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 59m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled up replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 4 to 5 59m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws 59m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Add eth0 [10.134.0.78/23] from ovn-kubernetes 59m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Container started 59m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Container created 59m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 59m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t 59m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 5 to 6 59m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Container started 59m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Add eth0 [10.133.0.71/23] from ovn-kubernetes 59m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Container created 59m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused... 59m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused 59m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Startup probe failed: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused 59m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Startup probe error: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused... 58m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 4 to 6 58m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 5 to 8 58m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp 58m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Add eth0 [10.132.0.79/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Add eth0 [10.134.0.80/23] from ovn-kubernetes 58m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt 58m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Add eth0 [10.133.0.72/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm 58m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Container started 58m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Container started 58m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Container created 58m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Add eth0 [10.132.0.80/23] from ovn-kubernetes 58m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Container started 58m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Add eth0 [10.133.0.73/23] from ovn-kubernetes 58m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Container created 58m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Container started 58m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Container created 58m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Container created 58m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Container started 58m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Container created 58m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused... 58m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Startup probe error: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused... 58m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Startup probe failed: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 58m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Readiness probe error: Get "http://10.134.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 58m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Readiness probe failed: Get "http://10.134.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 58m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc (combined from similar events): Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Created pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s 58m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled up replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 8 to 10 58m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm 58m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 6 to 7 57m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Container started 57m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Add eth0 [10.133.0.76/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Add eth0 [10.132.0.82/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Add eth0 [10.134.0.85/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Container created 57m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Container started 57m Normal Pulled pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Container created 57m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Container created 57m Normal Started pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Container started 57m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Startup probe error: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Startup probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Startup probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 57m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 57m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Startup probe failed: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 (combined from similar events): Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Created pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 57m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled up replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 5 to 10 57m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Add eth0 [10.133.0.77/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Add eth0 [10.133.0.78/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Container created 57m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Container started 57m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Container created 57m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Container started 57m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Add eth0 [10.132.0.83/23] from ovn-kubernetes 57m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Container started 57m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Container created 57m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Add eth0 [10.134.0.87/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Add eth0 [10.132.0.84/23] from ovn-kubernetes 57m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Container started 57m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Container created 57m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Pulled pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Container created 57m Normal Started pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Container started 57m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Startup probe failed: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused 57m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Startup probe failed: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Startup probe error: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused... 57m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Startup probe error: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused... 57m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Startup probe error: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Startup probe failed: Get "http://10.134.0.87:15021/healthz/ready": dial tcp 10.134.0.87:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 57m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 57m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c (combined from similar events): Created pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled up replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 7 to 10 56m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Add eth0 [10.133.0.81/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Container started 56m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Add eth0 [10.134.0.96/23] from ovn-kubernetes 56m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Container created 56m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Container started 56m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Container created 56m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Readiness probe failed: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Readiness probe error: Get "http://10.132.0.68:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Normal Started pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Container started 56m Normal Pulled pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Add eth0 [10.132.0.85/23] from ovn-kubernetes 56m Normal Created pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Container created 56m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Startup probe failed: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Startup probe error: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused... 56m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 3 to 4 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 56m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Container started 56m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Container created 56m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Add eth0 [10.134.0.99/23] from ovn-kubernetes 56m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Readiness probe error: Get "http://10.133.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Readiness probe failed: Get "http://10.133.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Readiness probe failed: Get "http://10.134.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Readiness probe failed: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Readiness probe failed: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Readiness probe error: Get "http://10.132.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Readiness probe error: Get "http://10.134.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Readiness probe error: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 56m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Startup probe failed: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Startup probe error: Get "http://10.134.0.99:15021/healthz/ready": dial tcp 10.134.0.99:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Startup probe failed: Get "http://10.134.0.99:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 56m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Startup probe error: Get "http://10.134.0.99:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Readiness probe failed: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Readiness probe error: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Add eth0 [10.132.0.88/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Add eth0 [10.132.0.87/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Container started 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j 55m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Add eth0 [10.133.0.87/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Container created 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k 55m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Add eth0 [10.134.0.107/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Container created 55m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 4 to 8 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 55m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg 55m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Container started 55m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Container started 55m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Container created 55m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Container created 55m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Container started 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Readiness probe error: Get "http://10.133.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Readiness probe failed: Get "http://10.133.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Readiness probe failed: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Readiness probe error: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Readiness probe failed: Get "http://10.134.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Readiness probe error: Get "http://10.134.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Readiness probe error: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Readiness probe error: Get "http://10.133.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Readiness probe failed: Get "http://10.134.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Readiness probe error: Get "http://10.134.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Readiness probe failed: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Readiness probe failed: Get "http://10.133.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Readiness probe error: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Readiness probe failed: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Readiness probe failed: Get "http://10.133.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Readiness probe error: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Readiness probe error: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Readiness probe error: Get "http://10.133.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Readiness probe failed: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Readiness probe error: Get "http://10.134.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Readiness probe failed: Get "http://10.134.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Readiness probe error: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Readiness probe failed: Get "http://10.132.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Readiness probe error: Get "http://10.132.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Readiness probe error: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Readiness probe failed: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Readiness probe error: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Readiness probe failed: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Readiness probe failed: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Readiness probe failed: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Readiness probe failed: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Readiness probe error: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Readiness probe failed: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Readiness probe failed: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Readiness probe error: Get "http://10.134.0.47:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Readiness probe error: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Readiness probe failed: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Readiness probe error: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Startup probe error: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Startup probe failed: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Readiness probe error: Get "http://10.134.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Startup probe error: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Readiness probe error: Get "http://10.132.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Readiness probe failed: Get "http://10.132.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Startup probe failed: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Readiness probe failed: Get "http://10.134.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 55m Normal AddedLabel gateway/e2e-ait-d4245aa3 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-d4245aa3 55m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Add eth0 [10.133.0.90/23] from ovn-kubernetes 55m Normal ScalingReplicaSet deployment/e2e-ait-d4245aa3-openshift-default Scaled up replica set e2e-ait-d4245aa3-openshift-default-5c5f469fb5 from 0 to 1 55m Normal SuccessfulCreate replicaset/e2e-ait-d4245aa3-openshift-default-5c5f469fb5 Created pod: e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 8 to 9 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v 55m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Container created 55m Normal AddedInterface pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Add eth0 [10.134.0.112/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Container started 55m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-8nrwm 55m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Add eth0 [10.134.0.113/23] from ovn-kubernetes 55m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 55m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 55m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-d4245aa3 Scaled up replica set payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587 from 0 to 1 55m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587 Created pod: payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx 55m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-d4245aa3 Scaled up replica set payload-processing-e2e-ait-d4245aa3-5ffd98cfc8 from 0 to 1 55m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8 Created pod: payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb 55m Warning FailedCreate replicaset/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8 Error creating: pods "payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-d4245aa3: serviceaccount "payload-processing-e2e-ait-d4245aa3" not found 55m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587 Error creating: pods "payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-d4245aa3: serviceaccount "payload-processing-e2e-ait-d4245aa3" not found 55m Normal Pulled pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Pulled pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Created pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Container created 55m Normal Started pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Container started 55m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.892s (2.892s including waiting). Image size: 185476477 bytes. 55m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Container created 55m Normal AddedInterface pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Add eth0 [10.132.0.90/23] from ovn-kubernetes 55m Normal Started pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Container started 55m Normal AddedInterface pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Add eth0 [10.132.0.91/23] from ovn-kubernetes 55m Normal Created pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Container created 55m Normal SuccessfulCreate replicaset/e2e-ait-c5e08e43-openshift-default-7fc9557759 Created pod: e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc 55m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Startup probe failed: Get "http://10.133.0.90:15021/healthz/ready": dial tcp 10.133.0.90:15021: connect: connection refused 55m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Container started 55m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Startup probe error: Get "http://10.133.0.90:15021/healthz/ready": dial tcp 10.133.0.90:15021: connect: connection refused... 55m Normal ScalingReplicaSet deployment/e2e-ait-c5e08e43-openshift-default Scaled up replica set e2e-ait-c5e08e43-openshift-default-7fc9557759 from 0 to 1 55m Normal AddedLabel gateway/e2e-ait-c5e08e43 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-c5e08e43 55m Normal AddedInterface pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Add eth0 [10.133.0.91/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 6 to 9 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 9; reason: cpu resource utilization (percentage of request) above target 55m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Readiness probe failed: Get "http://10.133.0.53:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Readiness probe error: Get "http://10.133.0.53:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Add eth0 [10.134.0.114/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Add eth0 [10.132.0.92/23] from ovn-kubernetes 55m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Readiness probe failed: Get "http://10.132.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Readiness probe error: Get "http://10.132.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Container created 55m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Container created 55m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Container started 55m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Add eth0 [10.133.0.93/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Container started 55m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Container created 55m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Container started 55m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-c5e08e43 Scaled up replica set payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b from 0 to 1 55m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-c5e08e43 Scaled up replica set payload-pre-processing-e2e-ait-c5e08e43-577c7df87f from 0 to 1 55m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b Created pod: payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl 55m Warning FailedCreate replicaset/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b Error creating: pods "payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-c5e08e43: serviceaccount "payload-processing-e2e-ait-c5e08e43" not found 55m Normal AddedInterface pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Add eth0 [10.133.0.94/23] from ovn-kubernetes 55m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f Error creating: pods "payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-c5e08e43: serviceaccount "payload-processing-e2e-ait-c5e08e43" not found 55m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f Created pod: payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 55m Normal Pulled pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Started pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Container started 55m Normal Created pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Container created 55m Normal AddedInterface pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Add eth0 [10.132.0.94/23] from ovn-kubernetes 55m Normal Started pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Container started 55m Normal Created pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Container created 55m Normal Pulled pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Startup probe error: Get "http://10.134.0.114:15021/healthz/ready": dial tcp 10.134.0.114:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Startup probe failed: Get "http://10.134.0.114:15021/healthz/ready": dial tcp 10.134.0.114:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Startup probe failed: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Startup probe error: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Startup probe failed: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Startup probe error: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused... 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-d4245aa3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-d4245aa3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Readiness probe error: HTTP probe failed with statuscode: 503... 54m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Readiness probe failed: HTTP probe failed with statuscode: 503 54m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Readiness probe failed: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Readiness probe error: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Readiness probe failed: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Readiness probe error: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Readiness probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Readiness probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Readiness probe error: Get "http://10.132.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Readiness probe failed: Get "http://10.132.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 (combined from similar events): Created pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 54m Normal Killing pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np Stopping container istio-proxy 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled up replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 9 to 10 54m Normal SuccessfulDelete replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Deleted pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w3-249dd4-openshift-default Scaled down replica set e2e-worker-w3-249dd4-openshift-default-59f7849b69 from 2 to 1 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default New size: 1; reason: All metrics below target 54m Normal AddedInterface pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Add eth0 [10.132.0.96/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Readiness probe failed: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Readiness probe error: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Normal Created pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Container created 54m Normal Started pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Container started 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-c5e08e43-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-c5e08e43-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Normal Started pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Container started 54m Normal Pulled pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Container created 54m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Startup probe failed: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Startup probe error: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused... 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 (combined from similar events): Created pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled up replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 9 to 10 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal Killing pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Stopping container payload-processing 54m Normal Killing pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Stopping container payload-pre-processing 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-d4245aa3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Normal Pulled pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Add eth0 [10.134.0.115/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Container started 54m Normal Created pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Container created 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-d4245aa3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning Unhealthy pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Startup probe failed: Get "http://10.134.0.112:15021/healthz/ready": dial tcp 10.134.0.112:15021: connect: connection refused 54m Warning ProbeError pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Startup probe error: Get "http://10.134.0.112:15021/healthz/ready": dial tcp 10.134.0.112:15021: connect: connection refused... 54m Normal Killing pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j Stopping container istio-proxy 54m Normal ScalingReplicaSet deployment/e2e-migrate-f215f8b0-gw-openshift-default Scaled up replica set e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c from 0 to 1 54m Normal AddedLabel gateway/e2e-migrate-f215f8b0-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-f215f8b0-gw 54m Normal SuccessfulCreate replicaset/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c Created pod: e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx 54m Normal AddedInterface pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Add eth0 [10.132.0.97/23] from ovn-kubernetes 54m Warning Unhealthy pod/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx Readiness probe failed: timeout: failed to connect service "10.132.0.90:50052" within 1s: context deadline exceeded 54m Warning Unhealthy pod/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb Readiness probe failed: timeout: failed to connect service "10.132.0.91:50052" within 1s: context deadline exceeded 54m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Startup probe error: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Startup probe failed: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7 Error creating: pods "payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-f215f8b0: serviceaccount "payload-processing-e2e-migrate-f215f8b0" not found 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-f215f8b0 Scaled up replica set payload-processing-e2e-migrate-f215f8b0-749ddf5f56 from 0 to 1 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7 Created pod: payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-f215f8b0 Scaled up replica set payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7 from 0 to 1 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-f215f8b0-749ddf5f56 Created pod: payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb 54m Warning FailedCreate replicaset/payload-processing-e2e-migrate-f215f8b0-749ddf5f56 Error creating: pods "payload-processing-e2e-migrate-f215f8b0-749ddf5f56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-f215f8b0: serviceaccount "payload-processing-e2e-migrate-f215f8b0" not found 54m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Startup probe failed: Get "http://10.134.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Startup probe error: Get "http://10.134.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Normal AddedInterface pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Add eth0 [10.132.0.98/23] from ovn-kubernetes 54m Normal Started pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Container started 54m Normal Pulled pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Container created 54m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Add eth0 [10.134.0.117/23] from ovn-kubernetes 54m Normal Started pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Container started 54m Normal Pulled pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Container created 54m Normal AddedLabel gateway/e2e-mt-87b0dcce Added label istio.io/rev=openshift-gateway to gateway e2e-mt-87b0dcce 54m Normal Started pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Container started 54m Normal Pulled pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Container created 54m Normal Killing pod/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2 Stopping container payload-pre-processing 54m Normal SuccessfulCreate replicaset/e2e-mt-87b0dcce-openshift-default-7fc555fd6f Created pod: e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 54m Normal ScalingReplicaSet deployment/e2e-mt-87b0dcce-openshift-default Scaled up replica set e2e-mt-87b0dcce-openshift-default-7fc555fd6f from 0 to 1 54m Normal AddedInterface pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Add eth0 [10.134.0.123/23] from ovn-kubernetes 54m Normal Killing pod/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl Stopping container payload-processing 54m Warning ProbeError pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused... 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-c5e08e43-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-c5e08e43-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning Unhealthy pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-87b0dcce-6dc659767 Error creating: pods "payload-pre-processing-e2e-mt-87b0dcce-6dc659767-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-87b0dcce: serviceaccount "payload-processing-e2e-mt-87b0dcce" not found 54m Warning Unhealthy pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Readiness probe failed: HTTP probe failed with statuscode: 503 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-87b0dcce Scaled up replica set payload-pre-processing-e2e-mt-87b0dcce-6dc659767 from 0 to 1 54m Warning ProbeError pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Readiness probe error: HTTP probe failed with statuscode: 503... 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-87b0dcce-777cd968dd Created pod: payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-87b0dcce Scaled up replica set payload-processing-e2e-mt-87b0dcce-777cd968dd from 0 to 1 54m Warning FailedCreate replicaset/payload-processing-e2e-mt-87b0dcce-777cd968dd Error creating: pods "payload-processing-e2e-mt-87b0dcce-777cd968dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-87b0dcce: serviceaccount "payload-processing-e2e-mt-87b0dcce" not found 54m Normal Killing pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc Stopping container istio-proxy 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-87b0dcce-6dc659767 Created pod: payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 54m Normal Created pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Container created 54m Normal Pulled pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Add eth0 [10.133.0.103/23] from ovn-kubernetes 54m Normal Started pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Container started 54m Normal SuccessfulCreate replicaset/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9 Created pod: e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq 54m Normal AddedLabel gateway/e2e-derive-3bdb4fd2 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-3bdb4fd2 54m Normal ScalingReplicaSet deployment/e2e-derive-3bdb4fd2-openshift-default Scaled up replica set e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9 from 0 to 1 54m Normal Created pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Container created 54m Normal Started pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Container started 54m Normal AddedInterface pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Add eth0 [10.132.0.100/23] from ovn-kubernetes 54m Normal Pulled pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Add eth0 [10.132.0.99/23] from ovn-kubernetes 53m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-3bdb4fd2 Scaled up replica set payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff from 0 to 1 53m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-3bdb4fd2 Scaled up replica set payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b from 0 to 1 53m Warning FailedCreate replicaset/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b Error creating: pods "payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-3bdb4fd2: serviceaccount "payload-processing-e2e-derive-3bdb4fd2" not found 53m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff Error creating: pods "payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-3bdb4fd2: serviceaccount "payload-processing-e2e-derive-3bdb4fd2" not found 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-249dd4-openshift-default Scaled up replica set e2e-worker-w3-249dd4-openshift-default-59f7849b69 from 1 to 3 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Created pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Created pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt 53m Normal Pulled pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Add eth0 [10.133.0.105/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Add eth0 [10.134.0.124/23] from ovn-kubernetes 53m Normal AddedInterface pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Add eth0 [10.134.0.125/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff Created pod: payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb 53m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b Created pod: payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw 53m Normal Pulled pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Container created 53m Normal AddedInterface pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Add eth0 [10.132.0.101/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Container started 53m Normal Created pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Container created 53m Normal Started pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Container started 53m Normal Started pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Container started 53m Normal Pulled pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 53m Normal Created pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Container created 53m Normal Started pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Container started 53m Normal Created pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Container created 53m Warning ProbeError pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Startup probe error: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Startup probe failed: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Startup probe error: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Startup probe failed: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused 53m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-a56cb5-openshift-default Scaled up replica set e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c from 0 to 1 53m Normal AddedInterface pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Add eth0 [10.133.0.106/23] from ovn-kubernetes 53m Normal AddedLabel gateway/e2e-shared-a-w1-a56cb5 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-a56cb5 53m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c Created pod: e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 53m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-a56cb5 Scaled up replica set payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8 from 0 to 1 53m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8 Error creating: pods "payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5: serviceaccount "payload-processing-e2e-shared-a-w1-a56cb5" not found 53m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-a56cb5 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499 from 0 to 1 53m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499 Created pod: payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x 53m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5: serviceaccount "payload-processing-e2e-shared-a-w1-a56cb5" not found 53m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8 Created pod: payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 53m Normal Pulled pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 53m Normal Started pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Container started 53m Normal Created pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Container created 53m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Add eth0 [10.134.0.126/23] from ovn-kubernetes 53m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 53m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Container created 53m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Add eth0 [10.134.0.127/23] from ovn-kubernetes 53m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Container started 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-f215f8b0-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-f215f8b0-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning ProbeError pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Readiness probe error: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Readiness probe failed: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Normal Created pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Container created 53m Normal Pulled pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Container started 53m Normal Killing pod/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb Stopping container payload-processing 53m Normal Killing pod/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw Stopping container payload-pre-processing 53m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-54hm4 53m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 53m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Add eth0 [10.133.0.108/23] from ovn-kubernetes 53m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 53m Warning ProbeError pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-f215f8b0-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-f215f8b0-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.866s (2.866s including waiting). Image size: 185476477 bytes. 53m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Container created 53m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Container started 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-a56cb5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-a56cb5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Normal Killing pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx Stopping container istio-proxy 53m Normal Started pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Container started 53m Normal Created pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Container created 53m Normal Pulled pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Warning Unhealthy pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Readiness probe failed: Get "http://10.132.0.100:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Readiness probe error: Get "http://10.132.0.100:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Normal Pulled pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Container started 53m Normal Created pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Container created 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-87b0dcce-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-87b0dcce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning ProbeError pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Startup probe error: Get "http://10.134.0.123:15021/healthz/ready": dial tcp 10.134.0.123:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Startup probe failed: Get "http://10.134.0.123:15021/healthz/ready": dial tcp 10.134.0.123:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Startup probe failed: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused 53m Warning ProbeError pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused... 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-3bdb4fd2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-3bdb4fd2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 52m Normal Killing pod/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw Stopping container payload-processing 52m Normal Killing pod/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb Stopping container payload-pre-processing 52m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-f83282-openshift-default Scaled up replica set e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c from 0 to 1 52m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c Created pod: e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph 52m Normal AddedLabel gateway/e2e-shared-b-w1-f83282 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-f83282 52m Normal AddedInterface pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Add eth0 [10.132.0.104/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-3bdb4fd2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 52m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-3bdb4fd2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 52m Normal Started pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Container started 52m Normal Created pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Container created 52m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-f83282 Scaled up replica set payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b from 0 to 1 52m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Add eth0 [10.134.0.131/23] from ovn-kubernetes 52m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b Created pod: payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td 52m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-f83282 Scaled up replica set payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff from 0 to 1 52m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff Created pod: payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 52m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff Error creating: pods "payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-f83282: serviceaccount "payload-processing-e2e-shared-b-w1-f83282" not found 52m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b Error creating: pods "payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-f83282: serviceaccount "payload-processing-e2e-shared-b-w1-f83282" not found 52m Normal Pulled pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal Started pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Container started 52m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Container started 52m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Add eth0 [10.134.0.132/23] from ovn-kubernetes 52m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Container created 52m Normal Created pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Container created 52m Warning ProbeError pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Startup probe error: Get "http://10.133.0.106:15021/healthz/ready": dial tcp 10.133.0.106:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Startup probe failed: Get "http://10.133.0.106:15021/healthz/ready": dial tcp 10.133.0.106:15021: connect: connection refused 52m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-87b0dcce-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 52m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-87b0dcce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 52m Normal Killing pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq Stopping container istio-proxy 52m Normal AddedLabel gateway/e2e-shared-a-w2-6b7ccc Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-6b7ccc 52m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-6b7ccc-openshift-default Scaled up replica set e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 from 0 to 1 52m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 Created pod: e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf 52m Normal AddedInterface pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Add eth0 [10.132.0.105/23] from ovn-kubernetes 52m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899 Created pod: payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 52m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899 Error creating: pods "payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc: serviceaccount "payload-processing-e2e-shared-a-w2-6b7ccc" not found 52m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-6b7ccc Scaled up replica set payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899 from 0 to 1 52m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-6b7ccc Scaled up replica set payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6 from 0 to 1 52m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6 Created pod: payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk 52m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6 Error creating: pods "payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc: serviceaccount "payload-processing-e2e-shared-a-w2-6b7ccc" not found 52m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Add eth0 [10.132.0.106/23] from ovn-kubernetes 52m Normal Pulled pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal Created pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Container created 52m Normal Started pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Container started 52m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Add eth0 [10.132.0.107/23] from ovn-kubernetes 52m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Container created 52m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Container started 52m Normal SuccessfulCreate replicaset/e2e-mt-ccf8bd61-openshift-default-7f595f9845 Created pod: e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 52m Normal ScalingReplicaSet deployment/e2e-mt-ccf8bd61-openshift-default Scaled up replica set e2e-mt-ccf8bd61-openshift-default-7f595f9845 from 0 to 1 52m Normal AddedLabel gateway/e2e-mt-ccf8bd61 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-ccf8bd61 52m Normal AddedInterface pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Add eth0 [10.134.0.135/23] from ovn-kubernetes 52m Warning Unhealthy pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Readiness probe failed: Get "http://10.132.0.104:15021/healthz/ready": read tcp 10.132.0.2:38870->10.132.0.104:15021: read: connection reset by peer 52m Normal Pulled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Container started 52m Normal Created pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Container created 52m Warning ProbeError pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Readiness probe error: Get "http://10.132.0.104:15021/healthz/ready": read tcp 10.132.0.2:38870->10.132.0.104:15021: read: connection reset by peer... 52m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-ccf8bd61 Scaled up replica set payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff from 0 to 1 52m Warning FailedCreate replicaset/payload-processing-e2e-mt-ccf8bd61-78478dd5bb Error creating: pods "payload-processing-e2e-mt-ccf8bd61-78478dd5bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ccf8bd61: serviceaccount "payload-processing-e2e-mt-ccf8bd61" not found 52m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff Error creating: pods "payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ccf8bd61: serviceaccount "payload-processing-e2e-mt-ccf8bd61" not found 52m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-ccf8bd61-78478dd5bb Created pod: payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt 52m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-ccf8bd61 Scaled up replica set payload-processing-e2e-mt-ccf8bd61-78478dd5bb from 0 to 1 52m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff Created pod: payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww 52m Normal AddedInterface pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Add eth0 [10.134.0.138/23] from ovn-kubernetes 52m Normal Started pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Container started 52m Normal Started pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Container started 52m Normal Created pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Container created 52m Normal Created pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Container created 52m Normal Pulled pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Add eth0 [10.132.0.108/23] from ovn-kubernetes 52m Normal Pulled pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Warning ProbeError pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Startup probe error: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Startup probe failed: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Normal AddedLabel gateway/e2e-api-a-891d6b Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-891d6b 51m Normal ScalingReplicaSet deployment/e2e-api-a-891d6b-openshift-default Scaled up replica set e2e-api-a-891d6b-openshift-default-798fd7ccdf from 0 to 1 51m Normal SuccessfulCreate replicaset/e2e-api-a-891d6b-openshift-default-798fd7ccdf Created pod: e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns 51m Normal AddedInterface pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Add eth0 [10.132.0.109/23] from ovn-kubernetes 51m Normal Created pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Container created 51m Normal Started pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Container started 51m Normal Pulled pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst 51m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 0 to 1 51m Normal AddedLabel gateway/e2e-shared-b-w2-d9eb76 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-d9eb76 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-891d6b-6cd74ddc58 Created pod: payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc Created pod: payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc Error creating: pods "payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-891d6b: serviceaccount "payload-processing-e2e-api-a-891d6b" not found 51m Warning FailedCreate replicaset/payload-processing-e2e-api-a-891d6b-6cd74ddc58 Error creating: pods "payload-processing-e2e-api-a-891d6b-6cd74ddc58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-891d6b: serviceaccount "payload-processing-e2e-api-a-891d6b" not found 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-891d6b Scaled up replica set payload-processing-e2e-api-a-891d6b-6cd74ddc58 from 0 to 1 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-891d6b Scaled up replica set payload-pre-processing-e2e-api-a-891d6b-c57b68cdc from 0 to 1 51m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Add eth0 [10.134.0.142/23] from ovn-kubernetes 51m Normal Started pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Container started 51m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Add eth0 [10.134.0.143/23] from ovn-kubernetes 51m Normal Created pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Container created 51m Normal Pulled pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Add eth0 [10.134.0.144/23] from ovn-kubernetes 51m Normal Started pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Container started 51m Normal Created pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Container created 51m Normal Pulled pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5 Error creating: pods "payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76: serviceaccount "payload-processing-e2e-shared-b-w2-d9eb76" not found 51m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b Error creating: pods "payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76: serviceaccount "payload-processing-e2e-shared-b-w2-d9eb76" not found 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b Created pod: payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-d9eb76 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5 from 0 to 1 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-d9eb76 Scaled up replica set payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b from 0 to 1 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5 Created pod: payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq 51m Warning Unhealthy pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Startup probe failed: Get "http://10.132.0.105:15021/healthz/ready": dial tcp 10.132.0.105:15021: connect: connection refused 51m Warning ProbeError pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Startup probe error: Get "http://10.132.0.105:15021/healthz/ready": dial tcp 10.132.0.105:15021: connect: connection refused... 51m Normal Created pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Container created 51m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Add eth0 [10.133.0.112/23] from ovn-kubernetes 51m Normal Pulled pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Started pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Container started 51m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Add eth0 [10.132.0.111/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cd4bde-openshift-default Scaled up replica set e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c from 0 to 1 51m Normal AddedInterface pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Add eth0 [10.133.0.114/23] from ovn-kubernetes 51m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Container started 51m Normal AddedLabel gateway/e2e-shared-a-w0-cd4bde Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-cd4bde 51m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c Created pod: e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk 51m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Container created 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-cd4bde Scaled up replica set payload-processing-e2e-shared-a-w0-cd4bde-76679b9544 from 0 to 1 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f Error creating: pods "payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde: serviceaccount "payload-processing-e2e-shared-a-w0-cd4bde" not found 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-cd4bde Scaled up replica set payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f from 0 to 1 51m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544 Error creating: pods "payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde: serviceaccount "payload-processing-e2e-shared-a-w0-cd4bde" not found 51m Normal Pulled pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Add eth0 [10.134.0.145/23] from ovn-kubernetes 51m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Add eth0 [10.133.0.115/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544 Created pod: payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f Created pod: payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 51m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Container created 51m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Killing pod/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb Stopping container payload-processing 51m Normal Started pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Container started 51m Normal Created pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Container created 51m Normal Killing pod/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66 Stopping container payload-pre-processing 51m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Container started 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-6b7ccc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-6b7ccc-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Readiness probe failed: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Readiness probe error: Get "http://10.132.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Readiness probe failed: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Readiness probe error: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Normal Pulled pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Container created 50m Normal Started pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Container started 50m Normal Killing pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 Stopping container istio-proxy 50m Warning ProbeError pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused 50m Normal SuccessfulCreate replicaset/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9 Created pod: e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f 50m Normal ScalingReplicaSet deployment/e2e-api-a-092bc0-openshift-default Scaled up replica set e2e-api-a-092bc0-openshift-default-6dd9ddd9b9 from 0 to 1 50m Normal AddedLabel gateway/e2e-api-a-092bc0 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-092bc0 50m Normal AddedInterface pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Add eth0 [10.134.0.146/23] from ovn-kubernetes 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-ccf8bd61-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-ccf8bd61-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 50m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 1 to 4 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Add eth0 [10.132.0.115/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Container started 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Container started 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Container created 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Add eth0 [10.133.0.117/23] from ovn-kubernetes 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Container started 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Container created 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Container created 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Add eth0 [10.134.0.147/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-092bc0 Scaled up replica set payload-processing-e2e-api-a-092bc0-577cb9f7f4 from 0 to 1 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc Error creating: pods "payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-092bc0: serviceaccount "payload-processing-e2e-api-a-092bc0" not found 50m Warning FailedCreate replicaset/payload-processing-e2e-api-a-092bc0-577cb9f7f4 Error creating: pods "payload-processing-e2e-api-a-092bc0-577cb9f7f4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-092bc0: serviceaccount "payload-processing-e2e-api-a-092bc0" not found 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-092bc0 Scaled up replica set payload-pre-processing-e2e-api-a-092bc0-74c8458bdc from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-092bc0-577cb9f7f4 Created pod: payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc Created pod: payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Container created 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Container started 50m Normal Created pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Container created 50m Normal Started pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Container started 50m Normal Created pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Container created 50m Normal Started pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Container started 50m Normal Pulled pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Add eth0 [10.134.0.148/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Add eth0 [10.134.0.149/23] from ovn-kubernetes 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-891d6b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-891d6b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Normal Started pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Container started 50m Normal Created pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Container created 50m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Startup probe failed: Get "http://10.134.0.147:15021/healthz/ready": dial tcp 10.134.0.147:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Startup probe failed: Get "http://10.133.0.117:15021/healthz/ready": dial tcp 10.133.0.117:15021: connect: connection refused 50m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Startup probe error: Get "http://10.133.0.117:15021/healthz/ready": dial tcp 10.133.0.117:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Startup probe error: Get "http://10.134.0.147:15021/healthz/ready": dial tcp 10.134.0.147:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Startup probe error: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Startup probe failed: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 50m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Startup probe failed: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused 50m Warning ProbeError pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Startup probe error: Get "http://10.132.0.109:15021/healthz/ready": dial tcp 10.132.0.109:15021: connect: connection refused... 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 6; reason: All metrics below target 50m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd 50m Normal ScalingReplicaSet deployment/e2e-inf-a-f4ab4a-openshift-default Scaled up replica set e2e-inf-a-f4ab4a-openshift-default-688bb6f67f from 0 to 1 50m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 50m Normal SuccessfulCreate replicaset/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f Created pod: e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 50m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled down replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 10 to 6 50m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg Stopping container istio-proxy 50m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg 50m Normal AddedLabel gateway/e2e-inf-a-f4ab4a Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-f4ab4a 50m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 50m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd Stopping container istio-proxy 50m Normal AddedInterface pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Add eth0 [10.132.0.116/23] from ovn-kubernetes 50m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 Stopping container istio-proxy 50m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 Stopping container istio-proxy 50m Normal SuccessfulCreate replicaset/e2e-api-b-7d9e82-openshift-default-5d58d7498f Created pod: e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 50m Normal ScalingReplicaSet deployment/e2e-api-b-7d9e82-openshift-default Scaled up replica set e2e-api-b-7d9e82-openshift-default-5d58d7498f from 0 to 1 50m Normal AddedLabel gateway/e2e-api-b-7d9e82 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-7d9e82 50m Warning ProbeError pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Readiness probe error: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused... 50m Normal AddedInterface pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Add eth0 [10.132.0.117/23] from ovn-kubernetes 50m Warning Unhealthy pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Readiness probe failed: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused 50m Normal Started pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Container started 50m Normal Pulled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Container created 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-f4ab4a Scaled up replica set payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c from 0 to 1 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-f4ab4a Scaled up replica set payload-processing-e2e-inf-a-f4ab4a-58874dbb4d from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d Created pod: payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c Created pod: payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-ccf8bd61-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d Error creating: pods "payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f4ab4a: serviceaccount "payload-processing-e2e-inf-a-f4ab4a" not found 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c Error creating: pods "payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f4ab4a: serviceaccount "payload-processing-e2e-inf-a-f4ab4a" not found 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-ccf8bd61-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Normal Pulled pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Add eth0 [10.132.0.118/23] from ovn-kubernetes 50m Normal Pulled pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 4 to 8 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 8; reason: 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt 50m Normal AddedInterface pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Add eth0 [10.132.0.120/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq 50m Normal Created pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Container created 50m Normal Started pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Container started 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Container created 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Add eth0 [10.134.0.150/23] from ovn-kubernetes 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Container started 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedLabel gateway/e2e-shared-b-w0-287ea2 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-287ea2 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Container created 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Container started 50m Normal Created pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Container created 50m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled up replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 0 to 1 50m Normal Started pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Container started 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Add eth0 [10.133.0.118/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Container started 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Add eth0 [10.132.0.122/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Add eth0 [10.132.0.123/23] from ovn-kubernetes 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Container created 50m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Container started 50m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Container created 50m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Add eth0 [10.132.0.121/23] from ovn-kubernetes 50m Warning Unhealthy pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Startup probe failed: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused 50m Warning ProbeError pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Startup probe error: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused... 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-891d6b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-891d6b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled down replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 6 to 5 49m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Startup probe failed: Get "http://10.134.0.150:15021/healthz/ready": dial tcp 10.134.0.150:15021: connect: connection refused 49m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Startup probe error: Get "http://10.134.0.150:15021/healthz/ready": dial tcp 10.134.0.150:15021: connect: connection refused... 49m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 Stopping container istio-proxy 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 5; reason: All metrics below target 49m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 49m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Startup probe error: Get "http://10.133.0.118:15021/healthz/ready": dial tcp 10.133.0.118:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Startup probe failed: Get "http://10.133.0.118:15021/healthz/ready": dial tcp 10.133.0.118:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Startup probe failed: Get "http://10.132.0.122:15021/healthz/ready": dial tcp 10.132.0.122:15021: connect: connection refused 49m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Startup probe error: Get "http://10.132.0.122:15021/healthz/ready": dial tcp 10.132.0.122:15021: connect: connection refused... 49m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Startup probe error: Get "http://10.132.0.121:15021/healthz/ready": dial tcp 10.132.0.121:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Startup probe failed: Get "http://10.132.0.121:15021/healthz/ready": dial tcp 10.132.0.121:15021: connect: connection refused 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-7d9e82 Scaled up replica set payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-api-b-7d9e82-84d68b46b6 Error creating: pods "payload-processing-e2e-api-b-7d9e82-84d68b46b6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-7d9e82: serviceaccount "payload-processing-e2e-api-b-7d9e82" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f Created pod: payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f Error creating: pods "payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-7d9e82: serviceaccount "payload-processing-e2e-api-b-7d9e82" not found 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-7d9e82 Scaled up replica set payload-processing-e2e-api-b-7d9e82-84d68b46b6 from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-7d9e82-84d68b46b6 Created pod: payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 49m Normal Pulled pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Add eth0 [10.134.0.152/23] from ovn-kubernetes 49m Normal Started pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Container started 49m Normal Created pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Container created 49m Normal Pulled pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Add eth0 [10.134.0.151/23] from ovn-kubernetes 49m Normal Created pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Container created 49m Normal Started pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Container started 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 9; reason: All metrics below target 49m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Container started 49m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Add eth0 [10.133.0.119/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Container created 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695 Error creating: pods "payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2: serviceaccount "payload-processing-e2e-shared-b-w0-287ea2" not found 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-287ea2 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695 from 0 to 1 49m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p 49m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d 49m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 5; reason: All metrics below target 49m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled down replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 10 to 5 49m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d Stopping container istio-proxy 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695 Created pod: payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb 49m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-287ea2 Scaled up replica set payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d Created pod: payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq 49m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d Error creating: pods "payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2: serviceaccount "payload-processing-e2e-shared-b-w0-287ea2" not found 49m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Add eth0 [10.134.0.153/23] from ovn-kubernetes 49m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r Stopping container istio-proxy 49m Normal Started pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Container started 49m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p Stopping container istio-proxy 49m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 Stopping container istio-proxy 49m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Container started 49m Normal Created pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Container created 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Add eth0 [10.134.0.154/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Container created 49m Normal Pulled pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 Stopping container istio-proxy 49m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Startup probe error: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Startup probe failed: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused 49m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 49m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 Stopping container istio-proxy 49m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled down replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 10 to 6 49m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm 49m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 6; reason: All metrics below target 49m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 49m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 Stopping container istio-proxy 49m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm Stopping container istio-proxy 49m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 Stopping container istio-proxy 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning ProbeError pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Readiness probe error: Get "http://10.134.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Readiness probe failed: Get "http://10.134.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal Created pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Container created 49m Normal Pulled pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Container started 49m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Stopping container istio-proxy 49m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-db77w 49m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 6; reason: All metrics below target 49m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d Stopping container istio-proxy 49m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv Stopping container istio-proxy 49m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv 49m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Stopping container istio-proxy 49m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq 49m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled down replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 10 to 6 49m Warning Unhealthy pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Startup probe failed: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused 49m Warning ProbeError pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Startup probe error: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused... 49m Normal AddedLabel gateway/e2e-api-b-b625eb Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-b625eb 49m Normal ScalingReplicaSet deployment/e2e-api-b-b625eb-openshift-default Scaled up replica set e2e-api-b-b625eb-openshift-default-7df9dd974c from 0 to 1 49m Normal SuccessfulCreate replicaset/e2e-api-b-b625eb-openshift-default-7df9dd974c Created pod: e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq 49m Normal AddedInterface pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Add eth0 [10.132.0.125/23] from ovn-kubernetes 49m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Readiness probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 49m Normal AddedLabel gateway/e2e-inf-a-9c8a2a Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-9c8a2a 49m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w Readiness probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 49m Normal SuccessfulCreate replicaset/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7 Created pod: e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h 49m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Readiness probe error: Get "http://10.132.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq Readiness probe failed: Get "http://10.132.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal AddedInterface pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Add eth0 [10.132.0.126/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/e2e-inf-a-9c8a2a-openshift-default Scaled up replica set e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7 from 0 to 1 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-092bc0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-092bc0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-b625eb Scaled up replica set payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-api-b-b625eb-6466fb5d76 Error creating: pods "payload-processing-e2e-api-b-b625eb-6466fb5d76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-b625eb: serviceaccount "payload-processing-e2e-api-b-b625eb" not found 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-b625eb Scaled up replica set payload-processing-e2e-api-b-b625eb-6466fb5d76 from 0 to 1 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc Error creating: pods "payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-b625eb: serviceaccount "payload-processing-e2e-api-b-b625eb" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc Created pod: payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-b625eb-6466fb5d76 Created pod: payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 49m Normal Created pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Container created 49m Normal Started pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Container started 49m Normal AddedInterface pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Add eth0 [10.134.0.157/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Container created 49m Normal Started pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Container started 49m Normal Pulled pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Add eth0 [10.134.0.156/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-9c8a2a Scaled up replica set payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4 from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4 Error creating: pods "payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-9c8a2a: serviceaccount "payload-processing-e2e-inf-a-9c8a2a" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c Created pod: payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4 Created pod: payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-9c8a2a Scaled up replica set payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c from 0 to 1 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c Error creating: pods "payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-9c8a2a: serviceaccount "payload-processing-e2e-inf-a-9c8a2a" not found 49m Normal Started pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Container started 49m Normal Created pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Container created 49m Normal Pulled pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Add eth0 [10.132.0.128/23] from ovn-kubernetes 49m Normal Started pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Container started 49m Normal Created pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Container created 49m Normal Pulled pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Add eth0 [10.134.0.158/23] from ovn-kubernetes 49m Normal Killing pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Stopping container payload-processing 49m Normal Killing pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Stopping container payload-pre-processing 49m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg Stopping container istio-proxy 49m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled down replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 10 to 9 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 9; reason: All metrics below target 49m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-092bc0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-092bc0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning Unhealthy pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 48m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m 48m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k Stopping container istio-proxy 48m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 8; reason: All metrics below target 48m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh 48m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j 48m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k 48m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default Scaled down replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 9 to 5 48m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 5; reason: All metrics below target 48m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled down replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 10 to 8 48m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m Stopping container istio-proxy 48m Warning Unhealthy pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Readiness probe failed: Get "http://10.132.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Warning ProbeError pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Readiness probe error: Get "http://10.132.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Normal Created pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Container created 48m Normal Pulled pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Container started 48m Normal Created pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Container created 48m Normal Pulled pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Container started 48m Normal AddedLabel gateway/e2e-inf-b-8e6b67 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-8e6b67 48m Normal SuccessfulCreate replicaset/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5 Created pod: e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 48m Normal ScalingReplicaSet deployment/e2e-inf-b-8e6b67-openshift-default Scaled up replica set e2e-inf-b-8e6b67-openshift-default-5c89b78cd5 from 0 to 1 48m Normal AddedInterface pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Add eth0 [10.134.0.159/23] from ovn-kubernetes 48m Warning Unhealthy pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 48m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r 48m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled down replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 6 to 5 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 5; reason: All metrics below target 48m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Stopping container istio-proxy 48m Warning Unhealthy pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Startup probe failed: Get "http://10.132.0.116:15021/healthz/ready": dial tcp 10.132.0.116:15021: connect: connection refused 48m Warning ProbeError pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Startup probe error: Get "http://10.132.0.116:15021/healthz/ready": dial tcp 10.132.0.116:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Startup probe error: Get "http://10.132.0.117:15021/healthz/ready": dial tcp 10.132.0.117:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Startup probe failed: Get "http://10.132.0.117:15021/healthz/ready": dial tcp 10.132.0.117:15021: connect: connection refused 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-8e6b67 Scaled up replica set payload-processing-e2e-inf-b-8e6b67-865ddcdd9d from 0 to 1 48m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d Error creating: pods "payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-8e6b67: serviceaccount "payload-processing-e2e-inf-b-8e6b67" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d Created pod: payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c Created pod: payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c Error creating: pods "payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-8e6b67: serviceaccount "payload-processing-e2e-inf-b-8e6b67" not found 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-8e6b67 Scaled up replica set payload-pre-processing-e2e-inf-b-8e6b67-68584d454c from 0 to 1 48m Normal Pulled pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-f4ab4a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Started pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Container started 48m Normal Started pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Container started 48m Normal AddedInterface pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Add eth0 [10.134.0.161/23] from ovn-kubernetes 48m Normal Created pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Container created 48m Normal Pulled pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-f4ab4a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Created pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Container created 48m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Add eth0 [10.134.0.162/23] from ovn-kubernetes 48m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w Stopping container istio-proxy 48m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj Stopping container istio-proxy 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 5; reason: All metrics below target 48m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default Scaled down replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 8 to 5 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 5; reason: All metrics below target 48m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Readiness probe error: Get "http://10.133.0.48:15021/healthz/ready": read tcp 10.133.0.2:41934->10.133.0.48:15021: read: connection reset by peer... 48m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c 48m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w 48m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled down replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 6 to 5 48m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt 48m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r Readiness probe failed: Get "http://10.133.0.48:15021/healthz/ready": read tcp 10.133.0.2:41934->10.133.0.48:15021: read: connection reset by peer 48m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-f4ab4a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-f4ab4a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal Killing pod/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4 Stopping container payload-processing 48m Normal Killing pod/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj Stopping container payload-pre-processing 48m Normal ScalingReplicaSet deployment/e2e-api-a-d4cb13-openshift-default Scaled up replica set e2e-api-a-d4cb13-openshift-default-84756765b7 from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-api-a-d4cb13-openshift-default-84756765b7 Created pod: e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd 48m Normal AddedLabel gateway/e2e-api-a-d4cb13 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-d4cb13 48m Normal AddedInterface pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Add eth0 [10.132.0.130/23] from ovn-kubernetes 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-d4cb13-7f86b775cb Created pod: payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph 48m Warning FailedCreate replicaset/payload-processing-e2e-api-a-d4cb13-7f86b775cb Error creating: pods "payload-processing-e2e-api-a-d4cb13-7f86b775cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d4cb13: serviceaccount "payload-processing-e2e-api-a-d4cb13" not found 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-d4cb13 Scaled up replica set payload-processing-e2e-api-a-d4cb13-7f86b775cb from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk 48m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled up replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 1 to 4 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-d4cb13 Scaled up replica set payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc from 0 to 1 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc Error creating: pods "payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d4cb13: serviceaccount "payload-processing-e2e-api-a-d4cb13" not found 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc Created pod: payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 48m Normal Pulled pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Container started 48m Normal Pulled pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Container created 48m Normal Started pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Container started 48m Normal Created pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Container created 48m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Add eth0 [10.132.0.132/23] from ovn-kubernetes 48m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Add eth0 [10.133.0.124/23] from ovn-kubernetes 48m Normal Started pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Container started 48m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Container created 48m Normal AddedInterface pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Add eth0 [10.134.0.164/23] from ovn-kubernetes 48m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Add eth0 [10.132.0.131/23] from ovn-kubernetes 48m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Add eth0 [10.134.0.165/23] from ovn-kubernetes 48m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Container started 48m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Container created 48m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Container created 48m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Container started 48m Normal Started pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Container started 48m Normal Created pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Container created 48m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Readiness probe failed: Get "http://10.132.0.123:15021/healthz/ready": read tcp 10.132.0.2:53030->10.132.0.123:15021: read: connection reset by peer 48m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Readiness probe error: Get "http://10.132.0.123:15021/healthz/ready": read tcp 10.132.0.2:53030->10.132.0.123:15021: read: connection reset by peer... 48m Normal Pulled pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Container started 48m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Container created 48m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-b625eb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-b625eb-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Startup probe error: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Startup probe failed: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused 48m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Startup probe error: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Startup probe failed: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Startup probe failed: Get "http://10.132.0.132:15021/healthz/ready": dial tcp 10.132.0.132:15021: connect: connection refused 48m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Startup probe error: Get "http://10.132.0.132:15021/healthz/ready": dial tcp 10.132.0.132:15021: connect: connection refused... 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-7d9e82-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning Unhealthy pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Startup probe failed: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused 48m Warning FailedCreate replicaset/payload-processing-e2e-mt-ccf8bd61-78478dd5bb Error creating: pods "payload-processing-e2e-mt-ccf8bd61-78478dd5bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ccf8bd61: serviceaccount "payload-processing-e2e-mt-ccf8bd61" not found 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff Error creating: pods "payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ccf8bd61: serviceaccount "payload-processing-e2e-mt-ccf8bd61" not found 48m Warning ProbeError pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Startup probe error: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused... 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-ccf8bd61 Scaled up replica set payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-ccf8bd61 Scaled up replica set payload-processing-e2e-mt-ccf8bd61-78478dd5bb from 0 to 1 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-7d9e82-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff Created pod: payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-ccf8bd61-78478dd5bb Created pod: payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx 48m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Startup probe failed: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused 48m Normal AddedInterface pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Add eth0 [10.132.0.133/23] from ovn-kubernetes 48m Normal Started pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Container started 48m Normal Created pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Container created 48m Normal Created pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Container created 48m Normal Started pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Container started 48m Normal Pulled pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal AddedInterface pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Add eth0 [10.132.0.134/23] from ovn-kubernetes 48m Normal Pulled pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Startup probe error: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused... 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal AddedLabel gateway/e2e-inf-b-19f646 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-19f646 47m Normal SuccessfulCreate replicaset/e2e-inf-b-19f646-openshift-default-5bdfd85d65 Created pod: e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw 47m Normal ScalingReplicaSet deployment/e2e-inf-b-19f646-openshift-default Scaled up replica set e2e-inf-b-19f646-openshift-default-5bdfd85d65 from 0 to 1 47m Normal AddedInterface pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Add eth0 [10.132.0.136/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled up replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 4 to 7 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 7; reason: 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-19f646-779d5cff99 Created pod: payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-19f646 Scaled up replica set payload-pre-processing-e2e-inf-b-19f646-779d5cff99 from 0 to 1 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-19f646-779d5cff99 Error creating: pods "payload-pre-processing-e2e-inf-b-19f646-779d5cff99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-19f646: serviceaccount "payload-processing-e2e-inf-b-19f646" not found 47m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-19f646-6f8dc65d78 Error creating: pods "payload-processing-e2e-inf-b-19f646-6f8dc65d78-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-19f646: serviceaccount "payload-processing-e2e-inf-b-19f646" not found 47m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Add eth0 [10.133.0.126/23] from ovn-kubernetes 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-19f646-6f8dc65d78 Created pod: payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-19f646 Scaled up replica set payload-processing-e2e-inf-b-19f646-6f8dc65d78 from 0 to 1 47m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Add eth0 [10.134.0.166/23] from ovn-kubernetes 47m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Add eth0 [10.132.0.137/23] from ovn-kubernetes 47m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Container created 47m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Container started 47m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Container started 47m Normal AddedInterface pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Add eth0 [10.132.0.138/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Add eth0 [10.134.0.167/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Container created 47m Normal Created pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Container created 47m Normal Started pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Container started 47m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Container started 47m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Container created 47m Normal Started pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Container started 47m Normal Created pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Container created 47m Normal Killing pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6 Stopping container payload-processing 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-b625eb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-b625eb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Startup probe failed: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Startup probe error: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused... 47m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Startup probe error: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Startup probe failed: Get "http://10.132.0.137:15021/healthz/ready": dial tcp 10.132.0.137:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Startup probe error: Get "http://10.132.0.137:15021/healthz/ready": dial tcp 10.132.0.137:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Startup probe failed: Get "http://10.134.0.166:15021/healthz/ready": dial tcp 10.134.0.166:15021: connect: connection refused 47m Warning Unhealthy pod/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m Readiness probe failed: timeout: failed to connect service "10.134.0.148:50052" within 1s: context deadline exceeded 47m Warning Unhealthy pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Readiness probe failed: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 47m Normal Created pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Container created 47m Normal Pulled pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Warning ProbeError pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Readiness probe error: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 47m Normal Started pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Container started 47m Normal Created pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Container created 47m Normal Started pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Container started 47m Normal Pulled pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Warning Unhealthy pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Startup probe failed: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused 47m Warning ProbeError pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Startup probe error: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused... 47m Warning ProbeError pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Startup probe error: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Startup probe failed: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-9c8a2a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-9c8a2a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-8e6b67-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-8e6b67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-9c8a2a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-9c8a2a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-8e6b67-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-8e6b67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal Killing pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns Stopping container istio-proxy 46m Normal Started pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Container started 46m Normal Pulled pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Container created 46m Normal SuccessfulCreate replicaset/e2e-api-b-a7c3b0-openshift-default-7d48cb547f Created pod: e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw 46m Normal AddedLabel gateway/e2e-api-b-a7c3b0 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-a7c3b0 46m Normal ScalingReplicaSet deployment/e2e-api-b-a7c3b0-openshift-default Scaled up replica set e2e-api-b-a7c3b0-openshift-default-7d48cb547f from 0 to 1 46m Normal AddedInterface pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Add eth0 [10.132.0.142/23] from ovn-kubernetes 46m Normal Killing pod/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk Stopping container payload-pre-processing 46m Normal Killing pod/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9 Stopping container payload-processing 46m Warning ProbeError pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Startup probe error: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Startup probe failed: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-a7c3b0 Scaled up replica set payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-api-b-a7c3b0-6f69f77fb Error creating: pods "payload-processing-e2e-api-b-a7c3b0-6f69f77fb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-a7c3b0: serviceaccount "payload-processing-e2e-api-b-a7c3b0" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc Error creating: pods "payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-a7c3b0: serviceaccount "payload-processing-e2e-api-b-a7c3b0" not found 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-a7c3b0 Scaled up replica set payload-processing-e2e-api-b-a7c3b0-6f69f77fb from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc Created pod: payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr 46m Normal Pulled pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Container created 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-a7c3b0-6f69f77fb Created pod: payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 46m Normal Started pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Container started 46m Normal Pulled pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Add eth0 [10.134.0.171/23] from ovn-kubernetes 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Add eth0 [10.134.0.170/23] from ovn-kubernetes 46m Normal Created pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Container created 46m Normal Started pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Container started 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d4cb13-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d4cb13-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal Killing pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f Stopping container istio-proxy 46m Warning ProbeError pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Readiness probe error: Get "http://10.132.0.136:15021/healthz/ready": read tcp 10.132.0.2:44512->10.132.0.136:15021: read: connection reset by peer... 46m Warning Unhealthy pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Readiness probe failed: Get "http://10.132.0.136:15021/healthz/ready": read tcp 10.132.0.2:44512->10.132.0.136:15021: read: connection reset by peer 46m Normal Created pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Container created 46m Normal Started pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Container started 46m Normal Pulled pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Warning Unhealthy pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Startup probe failed: Get "http://10.132.0.136:15021/healthz/ready": dial tcp 10.132.0.136:15021: connect: connection refused 46m Warning ProbeError pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Startup probe error: Get "http://10.132.0.136:15021/healthz/ready": dial tcp 10.132.0.136:15021: connect: connection refused... 46m Normal Killing pod/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m Stopping container payload-pre-processing 46m Normal Killing pod/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8 Stopping container payload-processing 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d4cb13-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d4cb13-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal Killing pod/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98 Stopping container payload-pre-processing 46m Normal Killing pod/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx Stopping container payload-processing 46m Normal ScalingReplicaSet deployment/e2e-inf-a-f4ab4a-openshift-default Scaled up replica set e2e-inf-a-f4ab4a-openshift-default-688bb6f67f from 1 to 2 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-f4ab4a-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 46m Normal Pulled pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Add eth0 [10.134.0.173/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f Created pod: e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq 46m Normal Started pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Container started 46m Normal Created pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Container created 46m Warning ProbeError pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Startup probe error: Get "http://10.134.0.173:15021/healthz/ready": dial tcp 10.134.0.173:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Startup probe failed: Get "http://10.134.0.173:15021/healthz/ready": dial tcp 10.134.0.173:15021: connect: connection refused 45m Normal SuccessfulCreate replicaset/e2e-inf-a-5d92fa-openshift-default-86bc6d4665 Created pod: e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-19f646-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-19f646-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal AddedLabel gateway/e2e-inf-a-5d92fa Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-5d92fa 45m Normal ScalingReplicaSet deployment/e2e-inf-a-5d92fa-openshift-default Scaled up replica set e2e-inf-a-5d92fa-openshift-default-86bc6d4665 from 0 to 1 45m Normal AddedInterface pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Add eth0 [10.132.0.144/23] from ovn-kubernetes 45m Normal Killing pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 Stopping container istio-proxy 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f Created pod: payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn 45m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-5d92fa-5497549574 Error creating: pods "payload-processing-e2e-inf-a-5d92fa-5497549574-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-5d92fa: serviceaccount "payload-processing-e2e-inf-a-5d92fa" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-5d92fa-5497549574 Created pod: payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-5d92fa Scaled up replica set payload-processing-e2e-inf-a-5d92fa-5497549574 from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-5d92fa Scaled up replica set payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f from 0 to 1 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f Error creating: pods "payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-5d92fa: serviceaccount "payload-processing-e2e-inf-a-5d92fa" not found 45m Normal Pulled pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Add eth0 [10.134.0.175/23] from ovn-kubernetes 45m Normal Pulled pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Container created 45m Normal Started pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Container started 45m Normal AddedInterface pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Add eth0 [10.134.0.176/23] from ovn-kubernetes 45m Normal Created pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Container created 45m Normal Started pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Container started 45m Normal SuccessfulCreate replicaset/e2e-inf-a-08bcfe-openshift-default-6469675667 Created pod: e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h 45m Normal AddedLabel gateway/e2e-inf-a-08bcfe Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-08bcfe 45m Normal ScalingReplicaSet deployment/e2e-inf-a-08bcfe-openshift-default Scaled up replica set e2e-inf-a-08bcfe-openshift-default-6469675667 from 0 to 1 45m Normal AddedInterface pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Add eth0 [10.132.0.145/23] from ovn-kubernetes 45m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c Error creating: pods "payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-08bcfe: serviceaccount "payload-processing-e2e-inf-a-08bcfe" not found 45m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled down replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 9 to 6 45m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b 45m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-08bcfe Scaled up replica set payload-processing-e2e-inf-a-08bcfe-5454cf5dd from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-08bcfe-5454cf5dd Created pod: payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr 45m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-08bcfe-5454cf5dd Error creating: pods "payload-processing-e2e-inf-a-08bcfe-5454cf5dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-08bcfe: serviceaccount "payload-processing-e2e-inf-a-08bcfe" not found 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-08bcfe Scaled up replica set payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c Created pod: payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g 45m Normal Pulled pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Add eth0 [10.132.0.147/23] from ovn-kubernetes 45m Normal Started pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Container started 45m Normal Created pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Container created 45m Normal Created pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Container created 45m Normal Pulled pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Add eth0 [10.132.0.148/23] from ovn-kubernetes 45m Normal Started pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Container started 45m Warning Unhealthy pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Readiness probe failed: Get "http://10.132.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Warning ProbeError pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Readiness probe error: Get "http://10.132.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Normal Pulled pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Container started 45m Normal Created pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Container created 45m Normal Killing pod/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25 Stopping container payload-pre-processing 45m Normal Killing pod/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph Stopping container payload-processing 45m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default Scaled down replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 5 to 3 45m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph 45m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 3; reason: All metrics below target 45m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph Stopping container istio-proxy 45m Normal SuccessfulCreate replicaset/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7 Created pod: e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j 45m Normal Killing pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-inf-a-9c8a2a-openshift-default Scaled up replica set e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7 from 1 to 2 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-9c8a2a-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 45m Normal Killing pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-inf-a-614c02-openshift-default Scaled up replica set e2e-inf-a-614c02-openshift-default-5984c4975 from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-inf-a-614c02-openshift-default-5984c4975 Created pod: e2e-inf-a-614c02-openshift-default-5984c4975-frdxh 45m Warning ProbeError pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Startup probe error: Get "http://10.132.0.142:15021/healthz/ready": dial tcp 10.132.0.142:15021: connect: connection refused... 45m Normal Pulled pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Container created 45m Normal Started pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Container started 45m Normal AddedLabel gateway/e2e-inf-a-614c02 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-614c02 45m Normal AddedInterface pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Add eth0 [10.134.0.177/23] from ovn-kubernetes 45m Warning Unhealthy pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Startup probe failed: Get "http://10.132.0.142:15021/healthz/ready": dial tcp 10.132.0.142:15021: connect: connection refused 45m Normal AddedInterface pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Add eth0 [10.132.0.149/23] from ovn-kubernetes 45m Warning Unhealthy pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Readiness probe failed: HTTP probe failed with statuscode: 503 45m Warning ProbeError pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq Readiness probe error: HTTP probe failed with statuscode: 503... 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-a7c3b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-a7c3b0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Readiness probe failed: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 Readiness probe error: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-614c02 Scaled up replica set payload-processing-e2e-inf-a-614c02-6db95f5b6d from 0 to 1 45m Warning ProbeError pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Startup probe error: Get "http://10.134.0.177:15021/healthz/ready": dial tcp 10.134.0.177:15021: connect: connection refused... 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9 Created pod: payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf 45m Warning Unhealthy pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Startup probe failed: Get "http://10.134.0.177:15021/healthz/ready": dial tcp 10.134.0.177:15021: connect: connection refused 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9 Error creating: pods "payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-614c02: serviceaccount "payload-processing-e2e-inf-a-614c02" not found 45m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-614c02-6db95f5b6d Error creating: pods "payload-processing-e2e-inf-a-614c02-6db95f5b6d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-614c02: serviceaccount "payload-processing-e2e-inf-a-614c02" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-614c02-6db95f5b6d Created pod: payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-614c02 Scaled up replica set payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9 from 0 to 1 45m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Add eth0 [10.134.0.179/23] from ovn-kubernetes 45m Normal Started pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Container started 45m Normal Started pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Container started 45m Normal Created pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Container created 45m Normal Created pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Container created 45m Normal Pulled pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Add eth0 [10.134.0.180/23] from ovn-kubernetes 45m Normal Pulled pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657 Created pod: e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz 44m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-a7b7bc-openshift-default Scaled up replica set e2e-shared-a-w3-a7b7bc-openshift-default-848696d657 from 0 to 1 44m Normal AddedLabel gateway/e2e-shared-a-w3-a7b7bc Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-a7b7bc 44m Normal AddedInterface pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Add eth0 [10.134.0.181/23] from ovn-kubernetes 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd Created pod: payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 4; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-a7b7bc Scaled up replica set payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd from 0 to 1 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd Error creating: pods "payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc: serviceaccount "payload-processing-e2e-shared-a-w3-a7b7bc" not found 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f Error creating: pods "payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc: serviceaccount "payload-processing-e2e-shared-a-w3-a7b7bc" not found 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f Created pod: payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz 44m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled down replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 5 to 4 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 3; reason: All metrics below target 44m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled down replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 5 to 3 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-a7b7bc Scaled up replica set payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f from 0 to 1 44m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Stopping container istio-proxy 44m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Add eth0 [10.134.0.182/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Container created 44m Normal Started pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Container started 44m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Container created 44m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Add eth0 [10.134.0.183/23] from ovn-kubernetes 44m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Container started 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 3; reason: All metrics below target 44m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Readiness probe failed: HTTP probe failed with statuscode: 503 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj 44m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default Scaled down replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 5 to 3 44m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj Stopping container istio-proxy 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-a7c3b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-a7c3b0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal Created pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Container created 44m Normal Started pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Container started 44m Normal Pulled pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 44m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 Stopping container istio-proxy 44m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled down replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 6 to 5 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 5; reason: All metrics below target 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 3; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz 44m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled down replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 4 to 3 44m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Stopping container istio-proxy 44m Warning ProbeError pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Startup probe error: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Startup probe failed: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-5d92fa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal ScalingReplicaSet deployment/e2e-inf-b-469724-openshift-default Scaled up replica set e2e-inf-b-469724-openshift-default-6c499b47c7 from 0 to 1 44m Normal SuccessfulCreate replicaset/e2e-inf-b-469724-openshift-default-6c499b47c7 Created pod: e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-5d92fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal AddedLabel gateway/e2e-inf-b-469724 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-469724 44m Normal AddedInterface pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Add eth0 [10.133.0.133/23] from ovn-kubernetes 44m Normal AddedLabel gateway/e2e-inf-b-09feeb Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-09feeb 44m Normal SuccessfulCreate replicaset/e2e-inf-b-09feeb-openshift-default-77dd7c9c95 Created pod: e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m 44m Normal ScalingReplicaSet deployment/e2e-inf-b-09feeb-openshift-default Scaled up replica set e2e-inf-b-09feeb-openshift-default-77dd7c9c95 from 0 to 1 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-5d92fa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-5d92fa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal AddedInterface pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Add eth0 [10.133.0.135/23] from ovn-kubernetes 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-469724-9fd448b79 Error creating: pods "payload-pre-processing-e2e-inf-b-469724-9fd448b79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-469724: serviceaccount "payload-processing-e2e-inf-b-469724" not found 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-469724-5b9fbc46c8 Created pod: payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 44m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-469724-5b9fbc46c8 Error creating: pods "payload-processing-e2e-inf-b-469724-5b9fbc46c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-469724: serviceaccount "payload-processing-e2e-inf-b-469724" not found 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-469724 Scaled up replica set payload-pre-processing-e2e-inf-b-469724-9fd448b79 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-469724-9fd448b79 Created pod: payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-469724 Scaled up replica set payload-processing-e2e-inf-b-469724-5b9fbc46c8 from 0 to 1 44m Normal AddedInterface pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Add eth0 [10.134.0.184/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Add eth0 [10.134.0.185/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Container created 44m Normal Started pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Container started 44m Normal Started pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Container started 44m Normal Created pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Container created 44m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Readiness probe error: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Readiness probe failed: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-09feeb Scaled up replica set payload-pre-processing-e2e-inf-b-09feeb-787c67455f from 0 to 1 44m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-09feeb-6d654c9df6 Error creating: pods "payload-processing-e2e-inf-b-09feeb-6d654c9df6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-09feeb: serviceaccount "payload-processing-e2e-inf-b-09feeb" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-09feeb Scaled up replica set payload-processing-e2e-inf-b-09feeb-6d654c9df6 from 0 to 1 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-09feeb-787c67455f Error creating: pods "payload-pre-processing-e2e-inf-b-09feeb-787c67455f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-09feeb: serviceaccount "payload-processing-e2e-inf-b-09feeb" not found 44m Normal Created pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Container created 44m Normal Pulled pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Container started 44m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Add eth0 [10.134.0.186/23] from ovn-kubernetes 44m Normal Started pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Container started 44m Normal Created pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Container created 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-09feeb-787c67455f Created pod: payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb 44m Normal Pulled pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Add eth0 [10.133.0.137/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-09feeb-6d654c9df6 Created pod: payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd 43m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 43m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 Stopping container istio-proxy 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 3; reason: All metrics below target 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default New size: 4; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 43m Normal Created pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Container created 43m Normal Pulled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Readiness probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq Readiness probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 43m Normal Started pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Container started 43m Warning ProbeError pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Startup probe error: Get "http://10.132.0.145:15021/healthz/ready": dial tcp 10.132.0.145:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Startup probe failed: Get "http://10.132.0.145:15021/healthz/ready": dial tcp 10.132.0.145:15021: connect: connection refused 43m Normal Killing pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd Stopping container istio-proxy 43m Normal Pulled pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Killing pod/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2 Stopping container payload-processing 43m Normal Killing pod/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr Stopping container payload-pre-processing 43m Normal Created pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Container created 43m Normal Started pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Container started 43m Normal AddedLabel gateway/e2e-inf-b-e3504b Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-e3504b 43m Normal SuccessfulCreate replicaset/e2e-inf-b-e3504b-openshift-default-668d85bd4c Created pod: e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj 43m Normal AddedInterface pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Add eth0 [10.132.0.151/23] from ovn-kubernetes 43m Normal ScalingReplicaSet deployment/e2e-inf-b-e3504b-openshift-default Scaled up replica set e2e-inf-b-e3504b-openshift-default-668d85bd4c from 0 to 1 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-614c02-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-614c02-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Stopping container payload-pre-processing 43m Normal Killing pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Stopping container payload-processing 43m Warning ProbeError pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Startup probe error: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Startup probe failed: Get "http://10.132.0.149:15021/healthz/ready": dial tcp 10.132.0.149:15021: connect: connection refused 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9 Created pod: payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-e3504b Scaled up replica set payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9 from 0 to 1 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9 Error creating: pods "payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e3504b: serviceaccount "payload-processing-e2e-inf-b-e3504b" not found 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-e3504b Scaled up replica set payload-processing-e2e-inf-b-e3504b-6d6d876cb4 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-e3504b-6d6d876cb4 Created pod: payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj 43m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-e3504b-6d6d876cb4 Error creating: pods "payload-processing-e2e-inf-b-e3504b-6d6d876cb4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e3504b: serviceaccount "payload-processing-e2e-inf-b-e3504b" not found 43m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Add eth0 [10.134.0.188/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Container started 43m Normal Started pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Container started 43m Normal Created pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Container created 43m Normal Pulled pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Add eth0 [10.134.0.187/23] from ovn-kubernetes 43m Normal Created pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Container created 43m Normal Pulled pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk Stopping container istio-proxy 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 4; reason: All metrics below target 43m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs 43m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp 43m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled down replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 7 to 4 43m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk 43m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-f44f74-openshift-default Scaled up replica set e2e-shared-b-w3-f44f74-openshift-default-677cb5496f from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f Created pod: e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf 43m Normal AddedLabel gateway/e2e-shared-b-w3-f44f74 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-f44f74 43m Normal AddedInterface pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Add eth0 [10.132.0.153/23] from ovn-kubernetes 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-08bcfe-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-08bcfe-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal Killing pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 Stopping container istio-proxy 43m Normal Pulled pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Created pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Container created 43m Normal Started pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Container started 43m Normal Killing pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Stopping container istio-proxy 43m Warning Unhealthy pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Readiness probe failed: HTTP probe failed with statuscode: 503 43m Warning ProbeError pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq Readiness probe error: HTTP probe failed with statuscode: 503... 43m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 43m Warning Unhealthy pod/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 43m Warning Unhealthy pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Startup probe failed: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused 43m Warning ProbeError pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Startup probe error: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused... 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db Created pod: payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-f44f74 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c Created pod: payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w 42m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db Error creating: pods "payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74: serviceaccount "payload-processing-e2e-shared-b-w3-f44f74" not found 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c Error creating: pods "payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74: serviceaccount "payload-processing-e2e-shared-b-w3-f44f74" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-f44f74 Scaled up replica set payload-processing-e2e-shared-b-w3-f44f74-7f644d6db from 0 to 1 42m Normal Pulled pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Started pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Container started 42m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Add eth0 [10.132.0.154/23] from ovn-kubernetes 42m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Add eth0 [10.132.0.155/23] from ovn-kubernetes 42m Normal Killing pod/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd Stopping container payload-processing 42m Normal Created pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Container created 42m Normal Killing pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Stopping container payload-pre-processing 42m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Container started 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-614c02-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Container created 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-614c02-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-a7b7bc-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-a7b7bc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal Started pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Container started 42m Normal Created pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Container created 42m Normal Pulled pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Container created 42m Normal Pulled pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Container started 42m Normal Killing pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Stopping container payload-pre-processing 42m Normal Killing pod/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47 Stopping container payload-processing 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-09feeb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-09feeb-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-a7b7bc-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-a7b7bc-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-469724-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-469724-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning ProbeError pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Startup probe error: Get "http://10.133.0.135:15021/healthz/ready": dial tcp 10.133.0.135:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Startup probe failed: Get "http://10.133.0.135:15021/healthz/ready": dial tcp 10.133.0.135:15021: connect: connection refused 42m Warning Unhealthy pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Startup probe failed: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused 42m Warning ProbeError pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Startup probe error: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused... 42m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 42m Normal Killing pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw Stopping container istio-proxy 42m Normal SuccessfulCreate replicaset/e2e-inf-a-1f6cef-openshift-default-78555bd8f7 Created pod: e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz 42m Normal ScalingReplicaSet deployment/e2e-inf-a-1f6cef-openshift-default Scaled up replica set e2e-inf-a-1f6cef-openshift-default-78555bd8f7 from 0 to 1 42m Normal AddedLabel gateway/e2e-inf-a-1f6cef Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-1f6cef 42m Normal AddedInterface pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Add eth0 [10.132.0.158/23] from ovn-kubernetes 42m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-1f6cef-d7f56846b Error creating: pods "payload-processing-e2e-inf-a-1f6cef-d7f56846b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1f6cef: serviceaccount "payload-processing-e2e-inf-a-1f6cef" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-1f6cef Scaled up replica set payload-processing-e2e-inf-a-1f6cef-d7f56846b from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-1f6cef Scaled up replica set payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-1f6cef-d7f56846b Created pod: payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr 42m Normal Pulled pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Add eth0 [10.132.0.160/23] from ovn-kubernetes 42m Normal Killing pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 Stopping container istio-proxy 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f Error creating: pods "payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1f6cef: serviceaccount "payload-processing-e2e-inf-a-1f6cef" not found 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Add eth0 [10.132.0.159/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f Created pod: payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz 42m Normal Killing pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j Stopping container istio-proxy 42m Normal Created pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Container created 42m Normal Started pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Container started 42m Normal Created pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Container created 42m Normal Started pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Container started 42m Normal Killing pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Stopping container istio-proxy 42m Warning Unhealthy pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Readiness probe failed: HTTP probe failed with statuscode: 503 42m Warning ProbeError pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h Readiness probe error: HTTP probe failed with statuscode: 503... 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-469724-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-469724-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal Pulled pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Container started 41m Normal Created pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Container created 41m Normal Killing pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Stopping container payload-pre-processing 41m Normal Killing pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Stopping container payload-processing 41m Warning ProbeError pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Startup probe error: Get "http://10.132.0.151:15021/healthz/ready": dial tcp 10.132.0.151:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Startup probe failed: Get "http://10.132.0.151:15021/healthz/ready": dial tcp 10.132.0.151:15021: connect: connection refused 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-e3504b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-e3504b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Warning Unhealthy pod/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst 41m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled down replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 5 to 4 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-e3504b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-e3504b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t 41m Normal Killing pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w3-249dd4-openshift-default-59f7849b69 Deleted pod: e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-249dd4-openshift-default New size: 2; reason: All metrics below target 41m Normal ScalingReplicaSet deployment/e2e-worker-w3-249dd4-openshift-default Scaled down replica set e2e-worker-w3-249dd4-openshift-default-59f7849b69 from 3 to 2 41m Warning Unhealthy pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Readiness probe failed: HTTP probe failed with statuscode: 503 41m Warning ProbeError pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t Readiness probe error: HTTP probe failed with statuscode: 503... 41m Normal Started pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Container started 41m Normal Pulled pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Container created 41m Normal Killing pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Stopping container istio-proxy 41m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-6b7ccc-openshift-default Scaled up replica set e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 from 1 to 2 41m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-095f55-openshift-default Scaled up replica set e2e-shared-a-w4-095f55-openshift-default-6556fb64bc from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc Created pod: e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp 41m Normal AddedLabel gateway/e2e-shared-a-w4-095f55 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w4-095f55 41m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 Created pod: e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w2-6b7ccc-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 41m Normal AddedInterface pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Add eth0 [10.134.0.194/23] from ovn-kubernetes 41m Normal AddedInterface pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Add eth0 [10.132.0.165/23] from ovn-kubernetes 41m Normal Pulled pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Container started 41m Normal Created pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Container created 41m Warning Unhealthy pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Readiness probe failed: HTTP probe failed with statuscode: 503 41m Warning ProbeError pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw Readiness probe error: HTTP probe failed with statuscode: 503... 41m Warning Unhealthy pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Startup probe failed: Get "http://10.132.0.153:15021/healthz/ready": dial tcp 10.132.0.153:15021: connect: connection refused 41m Warning ProbeError pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Startup probe error: Get "http://10.132.0.153:15021/healthz/ready": dial tcp 10.132.0.153:15021: connect: connection refused... 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-f44f74-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-f44f74-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b Created pod: payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w4-095f55 Scaled up replica set payload-processing-e2e-shared-a-w4-095f55-55bc548f7b from 0 to 1 41m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b Error creating: pods "payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-095f55: serviceaccount "payload-processing-e2e-shared-a-w4-095f55" not found 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9 Error creating: pods "payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-095f55: serviceaccount "payload-processing-e2e-shared-a-w4-095f55" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9 Created pod: payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w4-095f55 Scaled up replica set payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9 from 0 to 1 41m Warning ProbeError pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Startup probe error: Get "http://10.134.0.194:15021/healthz/ready": dial tcp 10.134.0.194:15021: connect: connection refused... 41m Normal Created pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Container created 41m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Add eth0 [10.134.0.195/23] from ovn-kubernetes 41m Normal AddedInterface pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Add eth0 [10.134.0.196/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Container created 41m Warning Unhealthy pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Startup probe failed: Get "http://10.134.0.194:15021/healthz/ready": dial tcp 10.134.0.194:15021: connect: connection refused 41m Normal Started pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Container started 41m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Container started 40m Normal Created pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Container created 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-08bcfe-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal ScalingReplicaSet deployment/e2e-inf-a-08bcfe-openshift-default Scaled up replica set e2e-inf-a-08bcfe-openshift-default-6469675667 from 1 to 2 40m Normal SuccessfulCreate replicaset/e2e-inf-a-08bcfe-openshift-default-6469675667 Created pod: e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 40m Normal Pulled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Add eth0 [10.134.0.197/23] from ovn-kubernetes 40m Normal Started pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Container started 40m Normal ScalingReplicaSet deployment/e2e-inf-a-5d92fa-openshift-default Scaled up replica set e2e-inf-a-5d92fa-openshift-default-86bc6d4665 from 1 to 2 40m Normal SuccessfulCreate replicaset/e2e-inf-a-5d92fa-openshift-default-86bc6d4665 Created pod: e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-5d92fa-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal AddedInterface pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Add eth0 [10.134.0.198/23] from ovn-kubernetes 40m Normal Started pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Container started 40m Normal Pulled pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Container created 40m Normal Pulled pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Container created 40m Normal Started pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Container started 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1f6cef-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1f6cef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning ProbeError pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Startup probe error: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Startup probe failed: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Startup probe error: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Startup probe failed: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Startup probe error: Get "http://10.132.0.158:15021/healthz/ready": dial tcp 10.132.0.158:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Startup probe failed: Get "http://10.132.0.158:15021/healthz/ready": dial tcp 10.132.0.158:15021: connect: connection refused 40m Normal SuccessfulCreate replicaset/e2e-inf-a-614c02-openshift-default-5984c4975 Created pod: e2e-inf-a-614c02-openshift-default-5984c4975-mdsld 40m Normal ScalingReplicaSet deployment/e2e-inf-a-614c02-openshift-default Scaled up replica set e2e-inf-a-614c02-openshift-default-5984c4975 from 1 to 2 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-614c02-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal Started pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Container started 40m Normal Created pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Container created 40m Normal Pulled pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Add eth0 [10.133.0.146/23] from ovn-kubernetes 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1f6cef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1f6cef-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning Unhealthy pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Startup probe failed: Get "http://10.133.0.146:15021/healthz/ready": dial tcp 10.133.0.146:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Startup probe error: Get "http://10.133.0.146:15021/healthz/ready": dial tcp 10.133.0.146:15021: connect: connection refused... 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-f44f74-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-9efc72-openshift-default New size: 2; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w2-9efc72-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc from 3 to 2 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-f44f74-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc Deleted pod: e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s 40m Normal ScalingReplicaSet deployment/e2e-inf-b-576a62-openshift-default Scaled up replica set e2e-inf-b-576a62-openshift-default-5b665b79f9 from 0 to 1 40m Normal AddedLabel gateway/e2e-inf-b-576a62 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-576a62 40m Normal SuccessfulCreate replicaset/e2e-inf-b-576a62-openshift-default-5b665b79f9 Created pod: e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd 40m Normal AddedInterface pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Add eth0 [10.132.0.169/23] from ovn-kubernetes 40m Warning ProbeError pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Readiness probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s Readiness probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-576a62 Scaled up replica set payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9 from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-576a62-c5d8ffb64 Created pod: payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-576a62 Scaled up replica set payload-processing-e2e-inf-b-576a62-c5d8ffb64 from 0 to 1 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9 Error creating: pods "payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-576a62: serviceaccount "payload-processing-e2e-inf-b-576a62" not found 40m Normal AddedInterface pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Add eth0 [10.134.0.200/23] from ovn-kubernetes 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9 Created pod: payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-576a62-c5d8ffb64 Error creating: pods "payload-processing-e2e-inf-b-576a62-c5d8ffb64-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-576a62: serviceaccount "payload-processing-e2e-inf-b-576a62" not found 40m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Stopping container istio-proxy 40m Normal Pulled pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Container created 40m Normal Created pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Container created 40m Normal Started pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Container started 40m Normal SuccessfulDelete replicaset/e2e-worker-w0-c5449c-openshift-default-6847484469 Deleted pod: e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm 40m Normal Pulled pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Add eth0 [10.132.0.170/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/e2e-worker-w0-c5449c-openshift-default Scaled down replica set e2e-worker-w0-c5449c-openshift-default-6847484469 from 3 to 2 40m Normal Started pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Container started 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c5449c-openshift-default New size: 2; reason: All metrics below target 40m Warning ProbeError pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 40m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 4 to 6 40m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Add eth0 [10.134.0.201/23] from ovn-kubernetes 40m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Container started 40m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Container created 40m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Add eth0 [10.133.0.147/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Container created 40m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Container started 39m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Startup probe failed: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused 39m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Startup probe error: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Startup probe failed: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused 39m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Startup probe error: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused... 39m Normal AddedInterface pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Add eth0 [10.133.0.149/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-08bcfe-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 39m Normal ScalingReplicaSet deployment/e2e-inf-a-08bcfe-openshift-default Scaled up replica set e2e-inf-a-08bcfe-openshift-default-6469675667 from 2 to 3 39m Normal SuccessfulCreate replicaset/e2e-inf-a-08bcfe-openshift-default-6469675667 Created pod: e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt 39m Normal Started pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Container started 39m Normal Created pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Container created 39m Warning Unhealthy pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Startup probe failed: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Startup probe error: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused... 39m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 2; reason: All metrics below target 39m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Stopping container istio-proxy 39m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled down replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 3 to 2 39m Normal SuccessfulDelete replicaset/e2e-worker-w5-e67b84-openshift-default-787fb8b88c Deleted pod: e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c 39m Warning ProbeError pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Readiness probe error: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e67b84-openshift-default New size: 2; reason: All metrics below target 39m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Stopping container istio-proxy 39m Warning Unhealthy pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Readiness probe failed: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal ScalingReplicaSet deployment/e2e-worker-w5-e67b84-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w5-e67b84-openshift-default-787fb8b88c from 3 to 2 39m Normal Started pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Container started 39m Normal Pulled pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Container created 39m Warning ProbeError pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning Unhealthy pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Startup probe failed: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused 39m Warning ProbeError pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Startup probe error: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused... 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-870193-openshift-default Scaled up replica set e2e-shared-b-w4-870193-openshift-default-6bdcd57969 from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-870193-openshift-default-6bdcd57969 Created pod: e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t 39m Normal AddedLabel gateway/e2e-shared-b-w4-870193 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w4-870193 39m Normal AddedInterface pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Add eth0 [10.132.0.172/23] from ovn-kubernetes 39m Normal SuccessfulDelete replicaset/e2e-worker-w6-6e0c55-openshift-default-88b6596b5 Deleted pod: e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6e0c55-openshift-default New size: 2; reason: All metrics below target 39m Normal ScalingReplicaSet deployment/e2e-worker-w6-6e0c55-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-6e0c55-openshift-default-88b6596b5 from 3 to 2 39m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v Stopping container istio-proxy 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w4-870193 Scaled up replica set payload-processing-e2e-shared-b-w4-870193-6c847f86c8 from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w4-870193-6c847f86c8 Created pod: payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7 Error creating: pods "payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-870193: serviceaccount "payload-processing-e2e-shared-b-w4-870193" not found 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7 Created pod: payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w4-870193 Scaled up replica set payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7 from 0 to 1 39m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w4-870193-6c847f86c8 Error creating: pods "payload-processing-e2e-shared-b-w4-870193-6c847f86c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-870193: serviceaccount "payload-processing-e2e-shared-b-w4-870193" not found 39m Normal AddedInterface pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Add eth0 [10.134.0.203/23] from ovn-kubernetes 39m Normal Started pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Container started 39m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Add eth0 [10.134.0.202/23] from ovn-kubernetes 39m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Container started 39m Normal Created pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Container created 39m Normal Pulled pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Created pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Container created 39m Normal Killing pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Stopping container payload-processing 39m Normal Killing pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Stopping container payload-pre-processing 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Killing pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Stopping container istio-proxy 39m Normal Killing pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Stopping container istio-proxy 39m Warning Unhealthy pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Warning Unhealthy pod/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Normal Killing pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Stopping container payload-pre-processing 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 3; reason: All metrics below target 38m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled down replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 4 to 3 38m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv Stopping container istio-proxy 38m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv 38m Normal Killing pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf Stopping container istio-proxy 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Warning Unhealthy pod/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 38m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-a56cb5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-a56cb5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-09feeb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-09feeb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f54779779 from 10 to 8 38m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-s2nd8 38m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-x4r26 38m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: All metrics below target 38m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-x4r26 Stopping container istio-proxy 38m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 Stopping container istio-proxy 38m Normal Pulled pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Container created 38m Normal Started pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Container started 38m Warning Unhealthy pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Readiness probe failed: Get "http://10.132.0.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Warning ProbeError pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Readiness probe error: Get "http://10.132.0.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Warning Unhealthy pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Startup probe failed: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Startup probe error: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused... 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-576a62-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-576a62-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f54779779 from 8 to 7 37m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-cnvpn Stopping container istio-proxy 37m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 37m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-cnvpn 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-576a62-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-576a62-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Pulled pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Container created 37m Normal Started pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Container started 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-870193-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-870193-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning Unhealthy pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Startup probe failed: Get "http://10.132.0.172:15021/healthz/ready": dial tcp 10.132.0.172:15021: connect: connection refused 37m Warning ProbeError pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Startup probe error: Get "http://10.132.0.172:15021/healthz/ready": dial tcp 10.132.0.172:15021: connect: connection refused... 37m Normal Killing pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Stopping container payload-processing 37m Normal Killing pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Stopping container payload-pre-processing 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-870193-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-870193-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-5ac614-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 36m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j Stopping container istio-proxy 36m Normal SuccessfulDelete replicaset/e2e-worker-w1-5ac614-openshift-default-77488748d7 Deleted pod: e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j 36m Normal ScalingReplicaSet deployment/e2e-worker-w1-5ac614-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-5ac614-openshift-default-77488748d7 from 3 to 2 36m Warning Unhealthy pod/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Normal Killing pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 Stopping container istio-proxy 36m Normal AddedLabel gateway/e2e-ipp-a-54086b Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-54086b 36m Warning Unhealthy pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal ScalingReplicaSet deployment/e2e-ipp-a-54086b-openshift-default Scaled up replica set e2e-ipp-a-54086b-openshift-default-66b75895f7 from 0 to 1 36m Normal SuccessfulCreate replicaset/e2e-ipp-a-54086b-openshift-default-66b75895f7 Created pod: e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj 36m Warning ProbeError pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt Stopping container istio-proxy 36m Normal AddedInterface pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Add eth0 [10.132.0.181/23] from ovn-kubernetes 36m Normal Killing pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Stopping container payload-pre-processing 36m Normal Killing pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Stopping container payload-processing 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-54086b Scaled up replica set payload-processing-e2e-ipp-a-54086b-68dcf85c6b from 0 to 1 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-54086b-58fb858499 Error creating: pods "payload-pre-processing-e2e-ipp-a-54086b-58fb858499-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-54086b: serviceaccount "payload-processing-e2e-ipp-a-54086b" not found 36m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-54086b-68dcf85c6b Error creating: pods "payload-processing-e2e-ipp-a-54086b-68dcf85c6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-54086b: serviceaccount "payload-processing-e2e-ipp-a-54086b" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-54086b Scaled up replica set payload-pre-processing-e2e-ipp-a-54086b-58fb858499 from 0 to 1 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-54086b-58fb858499 Created pod: payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh 36m Normal Started pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Container started 36m Normal Created pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Container created 36m Normal Created pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Container created 36m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-54086b-68dcf85c6b Created pod: payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj 36m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Add eth0 [10.134.0.215/23] from ovn-kubernetes 36m Normal Pulled pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Add eth0 [10.134.0.216/23] from ovn-kubernetes 36m Normal Started pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Container started 36m Normal Killing pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Stopping container payload-pre-processing 36m Warning Unhealthy pod/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Normal Killing pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh Stopping container istio-proxy 36m Warning Unhealthy pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd Readiness probe failed: timeout: failed to connect service "10.133.0.137:50052" within 1s: context deadline exceeded 35m Normal Killing pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Stopping container istio-proxy 35m Normal AddedLabel gateway/e2e-shared-a-w5-ec2fe7 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-ec2fe7 35m Normal Killing pod/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj Stopping container payload-processing 35m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 Created pod: e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r 35m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-ec2fe7-openshift-default Scaled up replica set e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 from 0 to 1 35m Normal Killing pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Stopping container payload-pre-processing 35m Normal AddedInterface pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Add eth0 [10.132.0.182/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-f83282-openshift-default Scaled up replica set e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c from 1 to 2 35m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c Created pod: e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Warning Unhealthy pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal Started pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Container started 35m Normal Created pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Container created 35m Warning ProbeError pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal AddedInterface pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Add eth0 [10.133.0.179/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946 Error creating: pods "payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7: serviceaccount "payload-processing-e2e-shared-a-w5-ec2fe7" not found 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-ec2fe7 Scaled up replica set payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff from 0 to 1 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-ec2fe7 Scaled up replica set payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946 from 0 to 1 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff Error creating: pods "payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7: serviceaccount "payload-processing-e2e-shared-a-w5-ec2fe7" not found 35m Normal Pulled pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Add eth0 [10.134.0.221/23] from ovn-kubernetes 35m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Container created 35m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Container started 35m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Add eth0 [10.134.0.220/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff Created pod: payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j 35m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946 Created pod: payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h 35m Normal Started pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Container started 35m Normal Created pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Container created 35m Warning Unhealthy pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Startup probe failed: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused 35m Warning ProbeError pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Startup probe error: Get "http://10.133.0.179:15021/healthz/ready": dial tcp 10.133.0.179:15021: connect: connection refused... 35m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Normal Killing pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Stopping container payload-pre-processing 35m Warning FailedToUpdateEndpointSlices service/e2e-inf-b-e3504b-openshift-default Error updating Endpoint Slices for Service openshift-ingress/e2e-inf-b-e3504b-openshift-default: failed to delete e2e-inf-b-e3504b-openshift-default-cmss6 EndpointSlice for Service openshift-ingress/e2e-inf-b-e3504b-openshift-default: endpointslices.discovery.k8s.io "e2e-inf-b-e3504b-openshift-default-cmss6" not found 35m Normal Killing pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Stopping container payload-processing 35m Normal Killing pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj Stopping container istio-proxy 35m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc Created pod: e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 35m Normal AddedLabel gateway/e2e-shared-a-w6-5250ee Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-5250ee 35m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-5250ee-openshift-default Scaled up replica set e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc from 0 to 1 35m Normal AddedInterface pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Add eth0 [10.132.0.184/23] from ovn-kubernetes 35m Normal Killing pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Stopping container istio-proxy 34m Normal Killing pod/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj Stopping container payload-processing 34m Normal Killing pod/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh Stopping container payload-pre-processing 34m Warning Unhealthy pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Readiness probe failed: HTTP probe failed with statuscode: 503 34m Warning ProbeError pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz Readiness probe error: HTTP probe failed with statuscode: 503... 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-5250ee-854db4f576 Created pod: payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c Created pod: payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c Error creating: pods "payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee: serviceaccount "payload-processing-e2e-shared-a-w6-5250ee" not found 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-5250ee Scaled up replica set payload-processing-e2e-shared-a-w6-5250ee-854db4f576 from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-5250ee Scaled up replica set payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c from 0 to 1 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-5250ee-854db4f576 Error creating: pods "payload-processing-e2e-shared-a-w6-5250ee-854db4f576-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee: serviceaccount "payload-processing-e2e-shared-a-w6-5250ee" not found 34m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Container created 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Add eth0 [10.132.0.185/23] from ovn-kubernetes 34m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Container started 34m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Pulled pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Container created 34m Normal Started pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Container started 34m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Add eth0 [10.132.0.186/23] from ovn-kubernetes 34m Warning Unhealthy pod/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning Unhealthy pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Readiness probe failed: Get "http://10.132.0.181:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Readiness probe error: Get "http://10.132.0.181:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal Killing pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj Stopping container payload-processing 34m Normal Started pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Container started 34m Normal Created pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Container created 34m Normal Pulled pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Container started 34m Normal Created pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Container created 34m Normal Pulled pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Warning ProbeError pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Startup probe error: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused... 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-54086b Scaled up replica set payload-processing-e2e-ipp-a-54086b-865c77b484 from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f Created pod: payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f Error creating: pods "payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-54086b: serviceaccount "payload-processing-e2e-ipp-a-54086b" not found 34m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-54086b-865c77b484 Error creating: pods "payload-processing-e2e-ipp-a-54086b-865c77b484-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-54086b: serviceaccount "payload-processing-e2e-ipp-a-54086b" not found 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-54086b-865c77b484 Created pod: payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-54086b Scaled up replica set payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f from 0 to 1 34m Warning Unhealthy pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Startup probe failed: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused 34m Warning Unhealthy pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Startup probe failed: Get "http://10.132.0.181:15021/healthz/ready": dial tcp 10.132.0.181:15021: connect: connection refused 34m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 34m Normal AddedInterface pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Add eth0 [10.132.0.187/23] from ovn-kubernetes 34m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Add eth0 [10.134.0.222/23] from ovn-kubernetes 34m Normal Pulling pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 34m Warning ProbeError pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Startup probe error: Get "http://10.132.0.181:15021/healthz/ready": dial tcp 10.132.0.181:15021: connect: connection refused... 34m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.684s (1.684s including waiting). Image size: 166612641 bytes. 34m Normal Created pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Container created 34m Normal Started pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Container started 34m Normal Started pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Container started 34m Normal Pulled pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.699s (1.699s including waiting). Image size: 166612641 bytes. 34m Normal Created pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Container created 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-54086b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-54086b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 Created pod: e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz 34m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-5a0a17-openshift-default Scaled up replica set e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 from 0 to 1 34m Normal AddedLabel gateway/e2e-shared-b-w5-5a0a17 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-5a0a17 34m Normal AddedInterface pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Add eth0 [10.132.0.188/23] from ovn-kubernetes 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-f83282-openshift-default Scaled up replica set e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c from 2 to 3 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 34m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c Created pod: e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-5a0a17 Scaled up replica set payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9 from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-5a0a17 Scaled up replica set payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8 from 0 to 1 34m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Container started 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8 Created pod: payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 34m Normal Started pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Container started 34m Normal Pulled pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Container created 34m Normal AddedInterface pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Add eth0 [10.134.0.223/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9 Created pod: payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Add eth0 [10.134.0.224/23] from ovn-kubernetes 34m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Container created 34m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Add eth0 [10.134.0.225/23] from ovn-kubernetes 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9 Error creating: pods "payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17: serviceaccount "payload-processing-e2e-shared-b-w5-5a0a17" not found 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8 Error creating: pods "payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17: serviceaccount "payload-processing-e2e-shared-b-w5-5a0a17" not found 34m Normal Created pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Container created 34m Normal Started pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Container started 34m Normal Pulled pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal ScalingReplicaSet deployment/e2e-ipp-b-852709-openshift-default Scaled up replica set e2e-ipp-b-852709-openshift-default-557fdb8995 from 0 to 1 34m Normal SuccessfulCreate replicaset/e2e-ipp-b-852709-openshift-default-557fdb8995 Created pod: e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt 34m Normal AddedLabel gateway/e2e-ipp-b-852709 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-852709 34m Normal AddedInterface pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Add eth0 [10.132.0.189/23] from ovn-kubernetes 34m Warning ProbeError pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Startup probe error: Get "http://10.134.0.223:15021/healthz/ready": dial tcp 10.134.0.223:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Startup probe failed: Get "http://10.134.0.223:15021/healthz/ready": dial tcp 10.134.0.223:15021: connect: connection refused 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-54086b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-54086b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-852709 Scaled up replica set payload-pre-processing-e2e-ipp-b-852709-67df66465f from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-852709 Scaled up replica set payload-processing-e2e-ipp-b-852709-bbff87ff6 from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-852709-67df66465f Created pod: payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-852709-67df66465f Error creating: pods "payload-pre-processing-e2e-ipp-b-852709-67df66465f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-852709: serviceaccount "payload-processing-e2e-ipp-b-852709" not found 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-852709-bbff87ff6 Created pod: payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 34m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-852709-bbff87ff6 Error creating: pods "payload-processing-e2e-ipp-b-852709-bbff87ff6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-852709: serviceaccount "payload-processing-e2e-ipp-b-852709" not found 34m Normal Started pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Container started 34m Normal Started pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Container started 34m Normal AddedInterface pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Add eth0 [10.134.0.227/23] from ovn-kubernetes 34m Normal Pulled pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Container created 34m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Add eth0 [10.134.0.226/23] from ovn-kubernetes 34m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Container created 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Normal Killing pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Stopping container istio-proxy 33m Warning Unhealthy pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 6 to 8 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 33m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Container started 33m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Add eth0 [10.134.0.230/23] from ovn-kubernetes 33m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Add eth0 [10.132.0.191/23] from ovn-kubernetes 33m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Container created 33m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Container created 33m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Container started 33m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Startup probe error: Get "http://10.132.0.191:15021/healthz/ready": dial tcp 10.132.0.191:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Startup probe failed: Get "http://10.132.0.191:15021/healthz/ready": dial tcp 10.132.0.191:15021: connect: connection refused 33m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Startup probe error: Get "http://10.134.0.230:15021/healthz/ready": dial tcp 10.134.0.230:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Startup probe failed: Get "http://10.134.0.230:15021/healthz/ready": dial tcp 10.134.0.230:15021: connect: connection refused 33m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499 Created pod: e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s 33m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-aae4d8-openshift-default Scaled up replica set e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499 from 0 to 1 33m Normal AddedLabel gateway/e2e-shared-b-w6-aae4d8 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-aae4d8 33m Normal AddedInterface pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Add eth0 [10.134.0.231/23] from ovn-kubernetes 33m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 33m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Stopping container istio-proxy 33m Normal ScalingReplicaSet deployment/e2e-worker-w4-b97379-openshift-default Scaled down replica set e2e-worker-w4-b97379-openshift-default-79f7b9f759 from 2 to 1 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-b97379-openshift-default New size: 1; reason: All metrics below target 33m Normal SuccessfulDelete replicaset/e2e-worker-w4-b97379-openshift-default-79f7b9f759 Deleted pod: e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb 33m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb Stopping container istio-proxy 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c Created pod: payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-aae4d8 Scaled up replica set payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-aae4d8 Scaled up replica set payload-processing-e2e-shared-b-w6-aae4d8-859d677588 from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-aae4d8-859d677588 Created pod: payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-aae4d8-859d677588 Error creating: pods "payload-processing-e2e-shared-b-w6-aae4d8-859d677588-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8: serviceaccount "payload-processing-e2e-shared-b-w6-aae4d8" not found 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c Error creating: pods "payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8: serviceaccount "payload-processing-e2e-shared-b-w6-aae4d8" not found 33m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Add eth0 [10.132.0.194/23] from ovn-kubernetes 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Add eth0 [10.132.0.193/23] from ovn-kubernetes 33m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Container created 33m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Container started 33m Normal Created pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Container created 33m Normal Started pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Container started 33m Normal Pulled pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Stopping container istio-proxy 33m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-g2h62 33m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-scx8v 33m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Stopping container discovery 33m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-nlrv8 33m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-8nrwm 33m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 33m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 Stopping container istio-proxy 33m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 33m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 33m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f54779779 from 7 to 4 33m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-scx8v Stopping container istio-proxy 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-g2h62 Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-scx8v Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-scx8v Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Readiness probe failed: Get "http://10.134.0.113:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 33m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm Readiness probe error: Get "http://10.134.0.113:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 33m Normal Created pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Container created 33m Normal Pulled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Started pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Container started 33m Normal Pulled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Container created 33m Normal Started pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Container started 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-aae4d8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-aae4d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning ProbeError pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Startup probe error: Get "http://10.132.0.184:15021/healthz/ready": dial tcp 10.132.0.184:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Startup probe failed: Get "http://10.132.0.184:15021/healthz/ready": dial tcp 10.132.0.184:15021: connect: connection refused 33m Warning Unhealthy pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Startup probe failed: Get "http://10.132.0.188:15021/healthz/ready": dial tcp 10.132.0.188:15021: connect: connection refused 33m Warning ProbeError pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Startup probe error: Get "http://10.132.0.188:15021/healthz/ready": dial tcp 10.132.0.188:15021: connect: connection refused... 33m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-a56cb5-openshift-default Scaled up replica set e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c from 1 to 2 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-a56cb5-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 33m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c Created pod: e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz 33m Normal Pulled pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Add eth0 [10.134.0.234/23] from ovn-kubernetes 33m Normal Created pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Container created 33m Normal Started pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Container started 33m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc Created pod: e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 33m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-095f55-openshift-default Scaled up replica set e2e-shared-a-w4-095f55-openshift-default-6556fb64bc from 1 to 2 32m Normal Pulled pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Container started 32m Normal Created pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Container created 32m Normal AddedInterface pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Add eth0 [10.134.0.235/23] from ovn-kubernetes 32m Warning ProbeError pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Startup probe error: Get "http://10.134.0.234:15021/healthz/ready": dial tcp 10.134.0.234:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Startup probe failed: Get "http://10.134.0.234:15021/healthz/ready": dial tcp 10.134.0.234:15021: connect: connection refused 32m Warning Unhealthy pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Startup probe failed: Get "http://10.134.0.235:15021/healthz/ready": dial tcp 10.134.0.235:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Startup probe error: Get "http://10.134.0.235:15021/healthz/ready": dial tcp 10.134.0.235:15021: connect: connection refused... 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-5a0a17-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-5a0a17-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-5250ee-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Killing pod/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2 Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft Stopping container payload-pre-processing 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-5250ee-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal SuccessfulCreate replicaset/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c Created pod: e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf 32m Normal AddedLabel gateway/e2e-ipp-a-ed2ede Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-ed2ede 32m Normal ScalingReplicaSet deployment/e2e-ipp-a-ed2ede-openshift-default Scaled up replica set e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c from 0 to 1 32m Normal AddedInterface pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Add eth0 [10.134.0.239/23] from ovn-kubernetes 32m Warning Unhealthy pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Readiness probe failed: Get "http://10.132.0.189:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Warning ProbeError pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Readiness probe error: Get "http://10.132.0.189:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 32m Normal Created pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Container created 32m Normal Started pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Container started 32m Normal Pulled pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 9; reason: cpu resource utilization (percentage of request) above target 32m Warning Unhealthy pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Startup probe failed: Get "http://10.132.0.189:15021/healthz/ready": dial tcp 10.132.0.189:15021: connect: connection refused 32m Warning ProbeError pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Startup probe error: Get "http://10.132.0.189:15021/healthz/ready": dial tcp 10.132.0.189:15021: connect: connection refused... 32m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default Scaled up replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 8 to 9 32m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c (combined from similar events): Created pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 32m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Created pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v 32m Normal Created pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Container created 32m Normal Pulled pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled up replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 2 to 3 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 32m Normal AddedInterface pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Add eth0 [10.133.0.189/23] from ovn-kubernetes 32m Normal Started pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Container started 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-852709-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-852709-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Pulled pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Container started 32m Normal AddedInterface pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Add eth0 [10.132.0.195/23] from ovn-kubernetes 32m Normal Created pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Container created 32m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ed2ede Scaled up replica set payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db from 0 to 1 32m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Startup probe error: Get "http://10.132.0.195:15021/healthz/ready": dial tcp 10.132.0.195:15021: connect: connection refused... 32m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Startup probe error: Get "http://10.133.0.189:15021/healthz/ready": dial tcp 10.133.0.189:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Startup probe failed: Get "http://10.133.0.189:15021/healthz/ready": dial tcp 10.133.0.189:15021: connect: connection refused 32m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Startup probe failed: Get "http://10.132.0.195:15021/healthz/ready": dial tcp 10.132.0.195:15021: connect: connection refused 32m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db Error creating: pods "payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ed2ede: serviceaccount "payload-processing-e2e-ipp-a-ed2ede" not found 32m Normal Started pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Container started 32m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95 Error creating: pods "payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ed2ede: serviceaccount "payload-processing-e2e-ipp-a-ed2ede" not found 32m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Normal Created pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Container created 32m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95 Created pod: payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 32m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ed2ede Scaled up replica set payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95 from 0 to 1 32m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Add eth0 [10.134.0.242/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db Created pod: payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 32m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Add eth0 [10.134.0.243/23] from ovn-kubernetes 32m Normal Started pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Container started 32m Normal Created pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Container created 32m Normal Pulled pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-ed2ede-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-ed2ede-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-852709-d58bf4df8 Created pod: payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs 32m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-852709-d58bf4df8 Error creating: pods "payload-processing-e2e-ipp-b-852709-d58bf4df8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-852709: serviceaccount "payload-processing-e2e-ipp-b-852709" not found 32m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-852709 Scaled up replica set payload-pre-processing-e2e-ipp-b-852709-545f78f65f from 0 to 1 32m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-852709 Scaled up replica set payload-processing-e2e-ipp-b-852709-d58bf4df8 from 0 to 1 32m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-852709-545f78f65f Created pod: payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 32m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-852709-545f78f65f Error creating: pods "payload-pre-processing-e2e-ipp-b-852709-545f78f65f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-852709: serviceaccount "payload-processing-e2e-ipp-b-852709" not found 32m Normal Created pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Container created 32m Normal AddedInterface pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Add eth0 [10.134.0.246/23] from ovn-kubernetes 32m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Add eth0 [10.134.0.245/23] from ovn-kubernetes 32m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 32m Normal Started pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Container started 32m Normal Pulled pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 32m Normal Created pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Container created 32m Normal Started pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Container started 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-852709-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-852709-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Normal Pulled pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Container created 31m Normal Started pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Container started 31m Warning Unhealthy pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Startup probe failed: Get "http://10.134.0.231:15021/healthz/ready": dial tcp 10.134.0.231:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Startup probe error: Get "http://10.134.0.231:15021/healthz/ready": dial tcp 10.134.0.231:15021: connect: connection refused... 31m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cd4bde-openshift-default Scaled up replica set e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c from 1 to 2 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 31m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c Created pod: e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml 31m Normal AddedInterface pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Add eth0 [10.134.0.250/23] from ovn-kubernetes 31m Normal Started pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Container started 31m Normal Pulled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Container created 31m Warning Unhealthy pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Startup probe failed: Get "http://10.134.0.250:15021/healthz/ready": dial tcp 10.134.0.250:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Startup probe error: Get "http://10.134.0.250:15021/healthz/ready": dial tcp 10.134.0.250:15021: connect: connection refused... 31m Normal AddedLabel gateway/e2e-ipp-a-1b36ad Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-1b36ad 31m Normal SuccessfulCreate replicaset/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c Created pod: e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd 31m Normal ScalingReplicaSet deployment/e2e-ipp-a-1b36ad-openshift-default Scaled up replica set e2e-ipp-a-1b36ad-openshift-default-7b78ff587c from 0 to 1 31m Normal AddedInterface pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Add eth0 [10.132.0.196/23] from ovn-kubernetes 31m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b Error creating: pods "payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1b36ad: serviceaccount "payload-processing-e2e-ipp-a-1b36ad" not found 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895 Error creating: pods "payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1b36ad: serviceaccount "payload-processing-e2e-ipp-a-1b36ad" not found 31m Normal Killing pod/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d Stopping container payload-processing 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-1b36ad Scaled up replica set payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b from 0 to 1 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895 Created pod: payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx 31m Normal Killing pod/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k Stopping container payload-pre-processing 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-1b36ad Scaled up replica set payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895 from 0 to 1 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b Created pod: payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf 31m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Add eth0 [10.133.0.191/23] from ovn-kubernetes 31m Normal Created pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Container created 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-aae4d8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal AddedInterface pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Add eth0 [10.132.0.197/23] from ovn-kubernetes 31m Normal Created pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Container created 31m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Container started 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-aae4d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal Started pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Container started 31m Normal Pulled pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 30m Normal Pulled pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Started pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Container started 30m Normal Created pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Container created 30m Warning Unhealthy pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Startup probe failed: Get "http://10.134.0.239:15021/healthz/ready": dial tcp 10.134.0.239:15021: connect: connection refused 30m Warning ProbeError pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Startup probe error: Get "http://10.134.0.239:15021/healthz/ready": dial tcp 10.134.0.239:15021: connect: connection refused... 30m Normal Killing pod/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477 Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766 Stopping container payload-pre-processing 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc Created pod: e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-5250ee-openshift-default Scaled up replica set e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc from 1 to 2 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-5250ee-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal Pulled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Add eth0 [10.134.0.254/23] from ovn-kubernetes 30m Normal Created pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Container created 30m Normal Started pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Container started 30m Normal Killing pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Stopping container istio-proxy 30m Warning ProbeError pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Startup probe error: Get "http://10.134.0.254:15021/healthz/ready": dial tcp 10.134.0.254:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Startup probe failed: Get "http://10.134.0.254:15021/healthz/ready": dial tcp 10.134.0.254:15021: connect: connection refused 30m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 Created pod: e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx 30m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-5a0a17-openshift-default Scaled up replica set e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 from 1 to 2 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-5a0a17-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal AddedInterface pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Add eth0 [10.134.1.1/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Container created 30m Normal Started pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Container started 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-ed2ede-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-ed2ede-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal AddedInterface pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Add eth0 [10.132.0.200/23] from ovn-kubernetes 30m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc Created pod: e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz 30m Normal AddedLabel gateway/e2e-ipp-a-bf314f Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-bf314f 30m Normal ScalingReplicaSet deployment/e2e-ipp-a-bf314f-openshift-default Scaled up replica set e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc from 0 to 1 30m Warning ProbeError pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Startup probe error: Get "http://10.134.1.1:15021/healthz/ready": dial tcp 10.134.1.1:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Startup probe failed: Get "http://10.134.1.1:15021/healthz/ready": dial tcp 10.134.1.1:15021: connect: connection refused 30m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ed2ede Scaled up replica set payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d from 0 to 1 30m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99 Created pod: payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z 30m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ed2ede Scaled up replica set payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99 from 0 to 1 30m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d Error creating: pods "payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ed2ede: serviceaccount "payload-processing-e2e-ipp-a-ed2ede" not found 30m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d Created pod: payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v 30m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99 Error creating: pods "payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ed2ede: serviceaccount "payload-processing-e2e-ipp-a-ed2ede" not found 29m Normal Created pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Container created 29m Normal Pulled pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-bf314f Scaled up replica set payload-processing-e2e-ipp-a-bf314f-fbf44d548 from 0 to 1 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7 Error creating: pods "payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf314f: serviceaccount "payload-processing-e2e-ipp-a-bf314f" not found 29m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Add eth0 [10.134.1.2/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-bf314f Scaled up replica set payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7 Created pod: payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-bf314f-fbf44d548 Error creating: pods "payload-processing-e2e-ipp-a-bf314f-fbf44d548-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf314f: serviceaccount "payload-processing-e2e-ipp-a-bf314f" not found 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-bf314f-fbf44d548 Created pod: payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Add eth0 [10.132.0.201/23] from ovn-kubernetes 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Add eth0 [10.134.1.3/23] from ovn-kubernetes 29m Normal Started pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Container started 29m Normal Started pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Container started 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Container started 29m Normal Created pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Container created 29m Normal AddedInterface pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Add eth0 [10.134.1.4/23] from ovn-kubernetes 29m Normal Created pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Container created 29m Normal Pulled pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Container started 29m Normal Created pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Container created 29m Normal Killing pod/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755 Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs Stopping container payload-processing 29m Warning Unhealthy pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Readiness probe failed: Get "http://10.132.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 29m Warning ProbeError pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Readiness probe error: Get "http://10.132.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 29m Normal Started pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Container started 29m Normal Created pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Container created 29m Normal Pulled pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Warning ProbeError pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Startup probe error: Get "http://10.132.0.196:15021/healthz/ready": dial tcp 10.132.0.196:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Startup probe failed: Get "http://10.132.0.196:15021/healthz/ready": dial tcp 10.132.0.196:15021: connect: connection refused 29m Normal SuccessfulCreate replicaset/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9 Created pod: e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl 29m Normal AddedLabel gateway/e2e-ipp-b-2afc11 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-2afc11 29m Normal ScalingReplicaSet deployment/e2e-ipp-b-2afc11-openshift-default Scaled up replica set e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9 from 0 to 1 29m Normal AddedInterface pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Add eth0 [10.133.0.192/23] from ovn-kubernetes 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-1b36ad-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-1b36ad-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-5250ee-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-5250ee-openshift-default Scaled up replica set e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc from 2 to 3 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc Created pod: e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc 29m Normal AddedInterface pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Add eth0 [10.133.0.193/23] from ovn-kubernetes 29m Normal Pulled pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Container created 29m Normal Started pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Container started 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-2afc11-7857797974 Error creating: pods "payload-processing-e2e-ipp-b-2afc11-7857797974-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2afc11: serviceaccount "payload-processing-e2e-ipp-b-2afc11" not found 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-2afc11-7857797974 Created pod: payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z 29m Normal AddedInterface pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Add eth0 [10.134.1.8/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-ec2fe7-openshift-default Scaled up replica set e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 from 1 to 2 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-2afc11 Scaled up replica set payload-processing-e2e-ipp-b-2afc11-7857797974 from 0 to 1 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 Created pod: e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf 29m Normal Pulled pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-2afc11 Scaled up replica set payload-pre-processing-e2e-ipp-b-2afc11-96d7db567 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567 Created pod: payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567 Error creating: pods "payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2afc11: serviceaccount "payload-processing-e2e-ipp-b-2afc11" not found 29m Normal AddedInterface pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Add eth0 [10.134.1.10/23] from ovn-kubernetes 29m Normal Created pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Container created 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Add eth0 [10.134.1.9/23] from ovn-kubernetes 29m Normal Created pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Container created 29m Normal Started pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Container started 29m Normal Pulled pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Container started 29m Normal Started pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Container started 29m Normal Created pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Container created 29m Normal Killing pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Stopping container istio-proxy 29m Normal Killing pod/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf Stopping container payload-processing 29m Normal AddedLabel gateway/e2e-ipp-b-59f9ed Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-59f9ed 29m Normal ScalingReplicaSet deployment/e2e-ipp-b-59f9ed-openshift-default Scaled up replica set e2e-ipp-b-59f9ed-openshift-default-6997fbc965 from 0 to 1 29m Normal SuccessfulCreate replicaset/e2e-ipp-b-59f9ed-openshift-default-6997fbc965 Created pod: e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 29m Warning ProbeError pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Startup probe error: Get "http://10.133.0.193:15021/healthz/ready": dial tcp 10.133.0.193:15021: connect: connection refused... 29m Normal AddedInterface pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Add eth0 [10.133.0.194/23] from ovn-kubernetes 29m Warning Unhealthy pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Startup probe failed: Get "http://10.133.0.193:15021/healthz/ready": dial tcp 10.133.0.193:15021: connect: connection refused 29m Warning ProbeError pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Startup probe error: Get "http://10.134.1.8:15021/healthz/ready": dial tcp 10.134.1.8:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Startup probe failed: Get "http://10.134.1.8:15021/healthz/ready": dial tcp 10.134.1.8:15021: connect: connection refused 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-59f9ed Scaled up replica set payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf from 0 to 1 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf Error creating: pods "payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-59f9ed: serviceaccount "payload-processing-e2e-ipp-b-59f9ed" not found 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6 Error creating: pods "payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-59f9ed: serviceaccount "payload-processing-e2e-ipp-b-59f9ed" not found 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-59f9ed Scaled up replica set payload-processing-e2e-ipp-b-59f9ed-59cc9599d6 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf Created pod: payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 29m Normal AddedInterface pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Add eth0 [10.132.0.203/23] from ovn-kubernetes 29m Normal Pulled pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6 Created pod: payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Add eth0 [10.132.0.202/23] from ovn-kubernetes 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Container created 29m Normal Created pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Container created 29m Normal Started pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Container started 29m Normal Started pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Container started 29m Normal Killing pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Stopping container payload-pre-processing 29m Normal AddedInterface pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Add eth0 [10.132.0.204/23] from ovn-kubernetes 29m Normal Created pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Container created 29m Normal Started pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Container started 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cd4bde-openshift-default Scaled up replica set e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c from 2 to 3 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c Created pod: e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj 29m Normal Pulled pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Warning ProbeError pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Startup probe error: Get "http://10.132.0.204:15021/healthz/ready": dial tcp 10.132.0.204:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Startup probe failed: Get "http://10.132.0.204:15021/healthz/ready": dial tcp 10.132.0.204:15021: connect: connection refused 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-1b36ad-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-1b36ad-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning Unhealthy pod/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-1b36ad Scaled up replica set payload-processing-e2e-ipp-a-1b36ad-7557679f96 from 0 to 1 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-1b36ad Scaled up replica set payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-1b36ad-7557679f96 Created pod: payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f Created pod: payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f Error creating: pods "payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1b36ad: serviceaccount "payload-processing-e2e-ipp-a-1b36ad" not found 28m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-1b36ad-7557679f96 Error creating: pods "payload-processing-e2e-ipp-a-1b36ad-7557679f96-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-1b36ad: serviceaccount "payload-processing-e2e-ipp-a-1b36ad" not found 28m Normal Started pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Container started 28m Normal Created pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Container created 28m Normal Created pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Container created 28m Normal Pulled pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 28m Normal Started pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Container started 28m Normal AddedInterface pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Add eth0 [10.132.0.206/23] from ovn-kubernetes 28m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Add eth0 [10.134.1.17/23] from ovn-kubernetes 28m Normal ScalingReplicaSet deployment/e2e-ipp-a-315ea2-openshift-default Scaled up replica set e2e-ipp-a-315ea2-openshift-default-656c6ccd4f from 0 to 1 28m Normal AddedLabel gateway/e2e-ipp-a-315ea2 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-315ea2 28m Normal SuccessfulCreate replicaset/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f Created pod: e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 28m Normal AddedInterface pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Add eth0 [10.134.1.18/23] from ovn-kubernetes 28m Normal Killing pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-r9r2z 28m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-r9r2z Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f54779779 from 4 to 3 28m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-ipp-a-bf314f-openshift-default Scaled up replica set e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc from 1 to 4 28m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc Created pod: e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb 28m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc Created pod: e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 28m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc Created pod: e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 28m Normal Started pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Container started 28m Normal AddedInterface pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Add eth0 [10.134.1.21/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Container created 28m Normal Pulled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Add eth0 [10.133.0.197/23] from ovn-kubernetes 28m Normal Created pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Container created 28m Normal Started pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Container started 28m Normal AddedInterface pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Add eth0 [10.133.0.196/23] from ovn-kubernetes 28m Normal Started pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Container started 28m Normal Created pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Container created 28m Normal Pulled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Readiness probe failed: Get "http://10.132.0.200:15021/healthz/ready": read tcp 10.132.0.2:47952->10.132.0.200:15021: read: connection reset by peer 28m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Readiness probe error: Get "http://10.132.0.200:15021/healthz/ready": read tcp 10.132.0.2:47952->10.132.0.200:15021: read: connection reset by peer... 28m Normal Started pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Container started 28m Normal Created pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Container created 28m Normal Pulled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Startup probe failed: Get "http://10.134.1.21:15021/healthz/ready": dial tcp 10.134.1.21:15021: connect: connection refused 28m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Startup probe error: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused... 28m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Startup probe error: Get "http://10.134.1.21:15021/healthz/ready": dial tcp 10.134.1.21:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Startup probe failed: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Startup probe failed: Get "http://10.133.0.196:15021/healthz/ready": dial tcp 10.133.0.196:15021: connect: connection refused 28m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Startup probe error: Get "http://10.133.0.196:15021/healthz/ready": dial tcp 10.133.0.196:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Startup probe failed: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused 28m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Startup probe error: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused... 28m Normal Killing pod/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv Stopping container payload-pre-processing 28m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4 Error creating: pods "payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-315ea2: serviceaccount "payload-processing-e2e-ipp-a-315ea2" not found 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4 Created pod: payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc 28m Normal Killing pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd Stopping container istio-proxy 28m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c Deleted pod: e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475 Created pod: payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-f83282-openshift-default New size: 2; reason: All metrics below target 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475 Error creating: pods "payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-315ea2: serviceaccount "payload-processing-e2e-ipp-a-315ea2" not found 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-315ea2 Scaled up replica set payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4 from 0 to 1 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-315ea2 Scaled up replica set payload-pre-processing-e2e-ipp-a-315ea2-5457f96475 from 0 to 1 28m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-f83282-openshift-default Scaled down replica set e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c from 3 to 2 28m Normal Pulled pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Add eth0 [10.134.1.24/23] from ovn-kubernetes 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal Started pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Container started 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal Started pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Container started 28m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal Created pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Container created 28m Normal AddedInterface pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Add eth0 [10.132.0.207/23] from ovn-kubernetes 28m Normal Created pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Container created 28m Normal Killing pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Stopping container payload-pre-processing 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Normal Killing pod/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69 Stopping container payload-pre-processing 28m Normal Killing pod/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z Stopping container payload-processing 27m Normal ScalingReplicaSet deployment/e2e-ipp-a-fe3e5e-openshift-default Scaled up replica set e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f Created pod: e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l 27m Normal AddedLabel gateway/e2e-ipp-a-fe3e5e Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-fe3e5e 27m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal AddedInterface pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Add eth0 [10.134.1.26/23] from ovn-kubernetes 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-bf314f-openshift-default New size: 5; reason: All metrics below target 27m Normal SuccessfulCreate replicaset/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc Created pod: e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g 27m Normal ScalingReplicaSet deployment/e2e-ipp-a-bf314f-openshift-default Scaled up replica set e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc from 4 to 5 27m Normal Pulled pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Add eth0 [10.132.0.208/23] from ovn-kubernetes 27m Normal Created pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Container created 27m Normal Started pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Container started 27m Normal Killing pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Stopping container istio-proxy 27m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Stopping container payload-processing 27m Warning ProbeError pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Startup probe error: Get "http://10.132.0.208:15021/healthz/ready": dial tcp 10.132.0.208:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Startup probe failed: Get "http://10.132.0.208:15021/healthz/ready": dial tcp 10.132.0.208:15021: connect: connection refused 27m Normal Killing pod/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7 Stopping container payload-pre-processing 27m Warning Unhealthy pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf Readiness probe error: HTTP probe failed with statuscode: 503... 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-bf314f Scaled up replica set payload-processing-e2e-ipp-a-bf314f-6dcd4559db from 0 to 1 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-bf314f Scaled up replica set payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-bf314f-6dcd4559db Created pod: payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9 Created pod: payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9 Error creating: pods "payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf314f: serviceaccount "payload-processing-e2e-ipp-a-bf314f" not found 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-bf314f-6dcd4559db Error creating: pods "payload-processing-e2e-ipp-a-bf314f-6dcd4559db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bf314f: serviceaccount "payload-processing-e2e-ipp-a-bf314f" not found 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Container created 27m Normal Pulled pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Add eth0 [10.132.0.209/23] from ovn-kubernetes 27m Normal Created pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Container created 27m Normal Started pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Add eth0 [10.134.1.28/23] from ovn-kubernetes 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7 Error creating: pods "payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e: serviceaccount "payload-processing-e2e-ipp-a-fe3e5e" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7 Created pod: payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d Created pod: payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-fe3e5e Scaled up replica set payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7 from 0 to 1 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d Error creating: pods "payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e: serviceaccount "payload-processing-e2e-ipp-a-fe3e5e" not found 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-fe3e5e Scaled up replica set payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d from 0 to 1 27m Normal Created pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Container created 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Container created 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Add eth0 [10.134.1.29/23] from ovn-kubernetes 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Add eth0 [10.134.1.30/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Container started 27m Normal Pulled pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Started pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Container started 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Killing pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Stopping container payload-processing 27m Warning ProbeError pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Readiness probe error: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused... 27m Normal Pulled pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Warning Unhealthy pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Readiness probe failed: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused 27m Normal Started pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Container started 27m Normal Created pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Container created 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-5a0a17-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 27m Normal Killing pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv Stopping container istio-proxy 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-5a0a17-openshift-default Scaled up replica set e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 from 2 to 3 27m Normal Killing pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 Stopping container istio-proxy 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 Created pod: e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j 27m Normal Killing pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc Stopping container istio-proxy 27m Normal Started pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Container started 27m Normal Created pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Container created 27m Normal Pulled pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Add eth0 [10.133.0.199/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Startup probe error: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Startup probe failed: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused 27m Warning Unhealthy pod/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal ScalingReplicaSet deployment/e2e-ipp-b-6e6016-openshift-default Scaled up replica set e2e-ipp-b-6e6016-openshift-default-5bd646478f from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-6e6016-openshift-default-5bd646478f Created pod: e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn 27m Normal AddedLabel gateway/e2e-ipp-b-6e6016 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-6e6016 27m Normal AddedInterface pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Add eth0 [10.132.0.211/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Startup probe error: Get "http://10.133.0.199:15021/healthz/ready": dial tcp 10.133.0.199:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Startup probe failed: Get "http://10.133.0.199:15021/healthz/ready": dial tcp 10.133.0.199:15021: connect: connection refused 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-59f9ed-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-59f9ed-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c Stopping container istio-proxy 27m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c 27m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default (combined from similar events): New size: 7; reason: All metrics below target 27m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 Stopping container istio-proxy 27m Normal Killing pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Stopping container payload-processing 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb Error creating: pods "payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2afc11: serviceaccount "payload-processing-e2e-ipp-b-2afc11" not found 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-2afc11 Scaled up replica set payload-pre-processing-e2e-ipp-b-2afc11-555679777f from 0 to 1 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-2afc11 Scaled up replica set payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb Created pod: payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff 27m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Stopping container payload-pre-processing 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-2afc11-555679777f Error creating: pods "payload-pre-processing-e2e-ipp-b-2afc11-555679777f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-2afc11: serviceaccount "payload-processing-e2e-ipp-b-2afc11" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-2afc11-555679777f Created pod: payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc 27m Normal Started pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Container created 27m Normal Started pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Add eth0 [10.134.1.31/23] from ovn-kubernetes 27m Normal AddedInterface pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Add eth0 [10.132.0.212/23] from ovn-kubernetes 27m Normal Pulled pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Container created 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Warning Unhealthy pod/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb Created pod: payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-6e6016 Scaled up replica set payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5 Created pod: payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb Error creating: pods "payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-6e6016: serviceaccount "payload-processing-e2e-ipp-b-6e6016" not found 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5 Error creating: pods "payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-6e6016: serviceaccount "payload-processing-e2e-ipp-b-6e6016" not found 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-6e6016 Scaled up replica set payload-processing-e2e-ipp-b-6e6016-8598cdcdbb from 0 to 1 27m Normal Started pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Container created 27m Normal AddedInterface pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Add eth0 [10.134.1.32/23] from ovn-kubernetes 27m Normal Pulled pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Container created 27m Normal Started pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Add eth0 [10.133.0.200/23] from ovn-kubernetes 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99 Created pod: payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-59f9ed Scaled up replica set payload-pre-processing-e2e-ipp-b-59f9ed-889559b99 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-59f9ed-5d66d88544 Created pod: payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-59f9ed Scaled up replica set payload-processing-e2e-ipp-b-59f9ed-5d66d88544 from 0 to 1 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99 Error creating: pods "payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-59f9ed: serviceaccount "payload-processing-e2e-ipp-b-59f9ed" not found 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-59f9ed-5d66d88544 Error creating: pods "payload-processing-e2e-ipp-b-59f9ed-5d66d88544-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-59f9ed: serviceaccount "payload-processing-e2e-ipp-b-59f9ed" not found 27m Normal Created pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Container created 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Add eth0 [10.134.1.33/23] from ovn-kubernetes 27m Normal Pulled pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Container started 27m Normal Created pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Container created 27m Normal Started pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Container created 27m Normal Pulled pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Add eth0 [10.132.0.213/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Container started 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-59f9ed-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-59f9ed-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning Unhealthy pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Startup probe failed: Get "http://10.133.0.192:15021/healthz/ready": dial tcp 10.133.0.192:15021: connect: connection refused 26m Warning ProbeError pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Startup probe error: Get "http://10.133.0.192:15021/healthz/ready": dial tcp 10.133.0.192:15021: connect: connection refused... 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-2afc11-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-2afc11-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Readiness probe failed: dial tcp 10.134.1.33:9004: connect: connection refused 26m Normal Started pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Container started 26m Normal Created pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Container created 26m Normal Pulled pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-315ea2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-315ea2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning ProbeError pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Startup probe error: Get "http://10.134.1.18:15021/healthz/ready": dial tcp 10.134.1.18:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Startup probe failed: Get "http://10.134.1.18:15021/healthz/ready": dial tcp 10.134.1.18:15021: connect: connection refused 26m Normal Killing pod/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz Stopping container payload-pre-processing 26m Normal Killing pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Stopping container istio-proxy 26m Warning ProbeError pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Readiness probe error: HTTP probe failed with statuscode: 503... 26m Warning Unhealthy pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s Readiness probe failed: HTTP probe failed with statuscode: 503 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-315ea2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-315ea2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-2afc11-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-2afc11-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Normal Killing pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Stopping container payload-processing 26m Normal Pulled pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Container created 26m Normal Started pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Container started 26m Warning Unhealthy pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Startup probe failed: Get "http://10.134.1.26:15021/healthz/ready": dial tcp 10.134.1.26:15021: connect: connection refused 26m Warning ProbeError pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Startup probe error: Get "http://10.134.1.26:15021/healthz/ready": dial tcp 10.134.1.26:15021: connect: connection refused... 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-fe3e5e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-fe3e5e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal Killing pod/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6 Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp Stopping container payload-pre-processing 26m Warning Unhealthy pod/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-315ea2-b47ff788d Error creating: pods "payload-processing-e2e-ipp-a-315ea2-b47ff788d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-315ea2: serviceaccount "payload-processing-e2e-ipp-a-315ea2" not found 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-315ea2 Scaled up replica set payload-processing-e2e-ipp-a-315ea2-b47ff788d from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-315ea2 Scaled up replica set payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9 from 0 to 1 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9 Error creating: pods "payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-315ea2: serviceaccount "payload-processing-e2e-ipp-a-315ea2" not found 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-315ea2-b47ff788d Created pod: payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l 25m Normal Started pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Container started 25m Normal AddedInterface pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Add eth0 [10.132.0.215/23] from ovn-kubernetes 25m Normal Pulled pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Add eth0 [10.132.0.214/23] from ovn-kubernetes 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9 Created pod: payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q 25m Normal Created pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Container created 25m Normal Created pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Container created 25m Normal Started pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Container started 25m Normal Killing pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt Stopping container istio-proxy 25m Normal Killing pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Stopping container istio-proxy 25m Warning Unhealthy pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Readiness probe failed: HTTP probe failed with statuscode: 503 25m Warning ProbeError pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s Readiness probe error: HTTP probe failed with statuscode: 503... 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-287ea2-openshift-default New size: 2; reason: All metrics below target 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-287ea2-openshift-default Scaled down replica set e2e-shared-b-w0-287ea2-openshift-default-76f4868567 from 3 to 2 25m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-287ea2-openshift-default-76f4868567 Deleted pod: e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v 25m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v Stopping container istio-proxy 25m Normal ScalingReplicaSet deployment/e2e-ipp-b-e430a0-openshift-default Scaled up replica set e2e-ipp-b-e430a0-openshift-default-645446d55f from 0 to 1 25m Normal AddedLabel gateway/e2e-ipp-b-e430a0 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-e430a0 25m Normal SuccessfulCreate replicaset/e2e-ipp-b-e430a0-openshift-default-645446d55f Created pod: e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 25m Normal Killing pod/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5 Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk Stopping container payload-pre-processing 25m Normal AddedInterface pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Add eth0 [10.134.1.39/23] from ovn-kubernetes 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-e430a0 Scaled up replica set payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-e430a0 Scaled up replica set payload-processing-e2e-ipp-b-e430a0-74fb694f6 from 0 to 1 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c Created pod: payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv 25m Normal Pulled pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-e430a0-74fb694f6 Created pod: payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-e430a0-74fb694f6 Error creating: pods "payload-processing-e2e-ipp-b-e430a0-74fb694f6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-e430a0: serviceaccount "payload-processing-e2e-ipp-b-e430a0" not found 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Add eth0 [10.132.0.216/23] from ovn-kubernetes 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal Created pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Container created 25m Normal Started pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Container started 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c Error creating: pods "payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-e430a0: serviceaccount "payload-processing-e2e-ipp-b-e430a0" not found 25m Normal Started pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Container started 25m Normal Created pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Container created 25m Normal Started pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Container started 25m Normal Created pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Container created 25m Normal Pulled pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Add eth0 [10.132.0.217/23] from ovn-kubernetes 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-6e6016-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-6e6016-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Normal Killing pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd Stopping container istio-proxy 25m Normal Killing pod/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9 Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6 Stopping container payload-processing 25m Warning ProbeError pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Startup probe error: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Startup probe failed: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-fe3e5e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-fe3e5e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-6e6016-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-6e6016-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d Error creating: pods "payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e: serviceaccount "payload-processing-e2e-ipp-a-fe3e5e" not found 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5 Error creating: pods "payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e: serviceaccount "payload-processing-e2e-ipp-a-fe3e5e" not found 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-fe3e5e Scaled up replica set payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-fe3e5e Scaled up replica set payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5 from 0 to 1 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d Created pod: payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz 25m Normal Pulled pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Add eth0 [10.132.0.219/23] from ovn-kubernetes 25m Normal Created pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Container created 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Add eth0 [10.132.0.218/23] from ovn-kubernetes 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5 Created pod: payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 25m Normal Created pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Container created 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 25m Normal Started pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Container started 25m Normal Started pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Container started 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-6e6016 Scaled up replica set payload-processing-e2e-ipp-b-6e6016-7f56db66b8 from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-6e6016 Scaled up replica set payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5 from 0 to 1 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Add eth0 [10.132.0.221/23] from ovn-kubernetes 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5 Error creating: pods "payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-6e6016: serviceaccount "payload-processing-e2e-ipp-b-6e6016" not found 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-6e6016-7f56db66b8 Created pod: payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5 Created pod: payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-6e6016-7f56db66b8 Error creating: pods "payload-processing-e2e-ipp-b-6e6016-7f56db66b8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-6e6016: serviceaccount "payload-processing-e2e-ipp-b-6e6016" not found 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Add eth0 [10.132.0.222/23] from ovn-kubernetes 24m Normal Created pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Container created 24m Normal Pulled pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Started pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Container started 24m Normal AddedLabel gateway/e2e-ipp-b-a76aa2 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-a76aa2 24m Normal Created pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Container created 24m Normal Started pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Container started 24m Normal Killing pod/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx Stopping container payload-pre-processing 24m Normal SuccessfulCreate replicaset/e2e-ipp-b-a76aa2-openshift-default-8588b5669 Created pod: e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 24m Normal Killing pod/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj Stopping container payload-processing 24m Normal AddedInterface pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Add eth0 [10.134.1.42/23] from ovn-kubernetes 24m Normal ScalingReplicaSet deployment/e2e-ipp-b-a76aa2-openshift-default Scaled up replica set e2e-ipp-b-a76aa2-openshift-default-8588b5669 from 0 to 1 24m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Stopping container istio-proxy 24m Normal Killing pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Stopping container istio-proxy 24m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Killing pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Stopping container istio-proxy 24m Warning ProbeError pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning ProbeError pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning Unhealthy pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Stopping container payload-processing 24m Normal Pulled pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Container created 24m Normal Started pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Container started 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b Error creating: pods "payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a76aa2: serviceaccount "payload-processing-e2e-ipp-b-a76aa2" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-a76aa2 Scaled up replica set payload-processing-e2e-ipp-b-a76aa2-79cd976c6b from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-a76aa2 Scaled up replica set payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5 Created pod: payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b Created pod: payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5 Error creating: pods "payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a76aa2: serviceaccount "payload-processing-e2e-ipp-b-a76aa2" not found 24m Normal Created pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Container created 24m Normal Pulled pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Add eth0 [10.134.1.47/23] from ovn-kubernetes 24m Normal Started pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Container started 24m Normal Started pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Container started 24m Normal Created pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Container created 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Add eth0 [10.134.1.46/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Warning Unhealthy pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Startup probe failed: Get "http://10.134.1.39:15021/healthz/ready": dial tcp 10.134.1.39:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Startup probe error: Get "http://10.134.1.39:15021/healthz/ready": dial tcp 10.134.1.39:15021: connect: connection refused... 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-e430a0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-e430a0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning Unhealthy pod/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l Stopping container payload-processing 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-e430a0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-e430a0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Normal Killing pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Stopping container istio-proxy 23m Normal Killing pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9 Stopping container payload-pre-processing 23m Normal Killing pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 Stopping container istio-proxy 23m Warning Unhealthy pod/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n Readiness probe failed: dial tcp 10.132.0.209:9004: connect: connection refused 23m Warning Unhealthy pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz Readiness probe error: HTTP probe failed with statuscode: 503... 23m Normal Killing pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g Stopping container istio-proxy 23m Normal Killing pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 Stopping container istio-proxy 23m Normal Killing pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz Stopping container istio-proxy 23m Normal Killing pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt Stopping container istio-proxy 23m Normal Killing pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb Stopping container istio-proxy 23m Normal Killing pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Stopping container payload-pre-processing 23m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v Stopping container payload-processing 23m Normal Killing pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml Stopping container istio-proxy 23m Normal Killing pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph Stopping container istio-proxy 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-cd4bde-openshift-default New size: 2; reason: All metrics below target 23m Normal Killing pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 Stopping container istio-proxy 23m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-cd4bde-openshift-default Scaled down replica set e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c from 3 to 2 23m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c Deleted pod: e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-e430a0 Scaled up replica set payload-processing-e2e-ipp-b-e430a0-779c85db98 from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc Created pod: payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-e430a0-779c85db98 Created pod: payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-e430a0 Scaled up replica set payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc from 0 to 1 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc Error creating: pods "payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-e430a0: serviceaccount "payload-processing-e2e-ipp-b-e430a0" not found 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-e430a0-779c85db98 Error creating: pods "payload-processing-e2e-ipp-b-e430a0-779c85db98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-e430a0: serviceaccount "payload-processing-e2e-ipp-b-e430a0" not found 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Add eth0 [10.134.1.61/23] from ovn-kubernetes 23m Normal Created pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Container created 23m Normal Started pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Container started 23m Normal AddedInterface pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Add eth0 [10.134.1.62/23] from ovn-kubernetes 23m Normal Pulled pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Container created 23m Normal Started pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Container started 23m Normal Killing pod/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz Stopping container payload-pre-processing 23m Normal Killing pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Stopping container payload-processing 23m Warning Unhealthy pod/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Normal Killing pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn Stopping container istio-proxy 23m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Normal Started pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Container started 23m Normal Created pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Container created 23m Normal Pulled pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Killing pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf Stopping container istio-proxy 22m Warning Unhealthy pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Startup probe failed: Get "http://10.134.1.42:15021/healthz/ready": dial tcp 10.134.1.42:15021: connect: connection refused 22m Normal Killing pod/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l Stopping container payload-processing 22m Warning ProbeError pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Startup probe error: Get "http://10.134.1.42:15021/healthz/ready": dial tcp 10.134.1.42:15021: connect: connection refused... 22m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96 Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s Stopping container payload-pre-processing 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-a76aa2 Scaled up replica set payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9 Error creating: pods "payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a76aa2: serviceaccount "payload-processing-e2e-ipp-b-a76aa2" not found 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d Error creating: pods "payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a76aa2: serviceaccount "payload-processing-e2e-ipp-b-a76aa2" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9 Created pod: payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-a76aa2 Scaled up replica set payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9 from 0 to 1 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d Created pod: payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff 22m Normal Killing pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Stopping container istio-proxy 22m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 22m Normal Killing pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj Stopping container istio-proxy 22m Normal Started pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Container started 22m Normal Pulled pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 22m Normal Created pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Container created 22m Normal Created pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Container created 22m Normal Started pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Container started 22m Normal AddedInterface pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Add eth0 [10.132.0.226/23] from ovn-kubernetes 22m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Add eth0 [10.132.0.225/23] from ovn-kubernetes 22m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg Stopping container istio-proxy 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-a76aa2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-a76aa2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Normal Killing pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj Stopping container istio-proxy 22m Warning ProbeError pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning Unhealthy pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx 22m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Stopping container istio-proxy 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 6; reason: All metrics below target 22m Normal Killing pod/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc Stopping container payload-pre-processing 22m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Readiness probe error: Get "http://10.132.0.191:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 22m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx Readiness probe failed: Get "http://10.132.0.191:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 22m Normal Killing pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl Stopping container istio-proxy 22m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Stopping container istio-proxy 22m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 22m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Stopping container istio-proxy 22m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-d9eb76-openshift-default (combined from similar events): Scaled down replica set e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c from 6 to 4 22m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c Deleted pod: e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-d9eb76-openshift-default New size: 4; reason: All metrics below target 22m Normal Killing pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Stopping container payload-pre-processing 22m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-6b7ccc-openshift-default Scaled down replica set e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 from 2 to 1 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w2-6b7ccc-openshift-default New size: 1; reason: All metrics below target 22m Normal SuccessfulDelete replicaset/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95 Deleted pod: e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf 22m Normal Killing pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf Stopping container istio-proxy 22m Normal Killing pod/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q Stopping container payload-pre-processing 22m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Readiness probe error: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused... 22m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Readiness probe error: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 Readiness probe failed: Get "http://10.134.0.201:15021/healthz/ready": dial tcp 10.134.0.201:15021: connect: connection refused 22m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc Readiness probe failed: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused 22m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Stopping container payload-pre-processing 22m Normal Killing pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j Stopping container istio-proxy 22m Normal SuccessfulDelete replicaset/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 Deleted pod: e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j 22m Normal Killing pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Stopping container payload-processing 22m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-5a0a17-openshift-default Scaled down replica set e2e-shared-b-w5-5a0a17-openshift-default-75d6966497 from 3 to 2 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-5a0a17-openshift-default New size: 2; reason: All metrics below target 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-ec2fe7-openshift-default New size: 1; reason: All metrics below target 22m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-ec2fe7-openshift-default Scaled down replica set e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 from 2 to 1 22m Normal Killing pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf Stopping container istio-proxy 22m Normal SuccessfulDelete replicaset/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5 Deleted pod: e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf 22m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 22m Normal Killing pod/maas-default-gateway-openshift-default-6f54779779-8k46t Stopping container istio-proxy 22m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f54779779 Deleted pod: maas-default-gateway-openshift-default-6f54779779-8k46t 22m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f54779779 from 3 to 2 22m Normal Killing pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Stopping container istio-proxy 22m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-8k46t Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-8k46t Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal Killing pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g Stopping container istio-proxy 21m Warning Unhealthy pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Readiness probe failed: Get "http://10.134.1.18:15021/healthz/ready": dial tcp 10.134.1.18:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 Readiness probe error: Get "http://10.134.1.18:15021/healthz/ready": dial tcp 10.134.1.18:15021: connect: connection refused... 21m Normal Killing pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 Stopping container istio-proxy 21m Normal SuccessfulDelete replicaset/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc Deleted pod: e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 21m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-095f55-openshift-default Scaled down replica set e2e-shared-a-w4-095f55-openshift-default-6556fb64bc from 2 to 1 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-095f55-openshift-default New size: 1; reason: All metrics below target 21m Warning Unhealthy pod/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal Killing pod/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4 Stopping container payload-pre-processing 21m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Stopping container istio-proxy 21m Normal Killing pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s Stopping container istio-proxy 21m Warning Unhealthy pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning ProbeError pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 Readiness probe error: HTTP probe failed with statuscode: 503... 21m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Stopping container payload-processing 21m Normal Killing pod/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Stopping container payload-pre-processing 21m Normal Killing pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 Stopping container istio-proxy 21m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Stopping container istio-proxy 21m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Stopping container istio-proxy 21m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Stopping container istio-proxy 21m Normal Killing pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Stopping container istio-proxy 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning ProbeError pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Killing pod/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff Stopping container payload-processing 21m Warning Unhealthy pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Killing pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Stopping container payload-processing 21m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 Stopping container istio-proxy 21m Normal Killing pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 Stopping container istio-proxy 21m Normal Killing pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Stopping container payload-processing 20m Normal Killing pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Stopping container payload-pre-processing 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp Stopping container istio-proxy 20m Warning Unhealthy pod/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Stopping container istio-proxy 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r Readiness probe error: HTTP probe failed with statuscode: 503... 20m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm Stopping container istio-proxy 20m Normal Killing pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 Stopping container istio-proxy 20m Normal Killing pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Stopping container payload-processing 20m Normal Killing pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz Stopping container istio-proxy 20m Warning Unhealthy pod/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t Stopping container istio-proxy 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Stopping container payload-pre-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt Stopping container istio-proxy 19m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-vv8z5 19m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 2 to 4 19m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 19m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-mpgxb 19m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Container created 19m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Add eth0 [10.132.0.229/23] from ovn-kubernetes 19m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Container started 19m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Add eth0 [10.134.1.82/23] from ovn-kubernetes 19m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Container started 19m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Container created 19m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Startup probe error: Get "http://10.134.1.82:15021/healthz/ready": dial tcp 10.134.1.82:15021: connect: connection refused... 19m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 Startup probe failed: Get "http://10.134.1.82:15021/healthz/ready": dial tcp 10.134.1.82:15021: connect: connection refused 19m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Startup probe failed: Get "http://10.132.0.229:15021/healthz/ready": dial tcp 10.132.0.229:15021: connect: connection refused 19m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-mpgxb Startup probe error: Get "http://10.132.0.229:15021/healthz/ready": dial tcp 10.132.0.229:15021: connect: connection refused... 19m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 19m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 4 to 5 19m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-czfvt 19m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-czfvt Container created 19m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-czfvt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-czfvt Add eth0 [10.134.1.84/23] from ovn-kubernetes 19m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-czfvt Container started 18m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-czfvt Startup probe error: Get "http://10.134.1.84:15021/healthz/ready": dial tcp 10.134.1.84:15021: connect: connection refused... 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-czfvt Startup probe failed: Get "http://10.134.1.84:15021/healthz/ready": dial tcp 10.134.1.84:15021: connect: connection refused 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-xljwk 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f54779779 from 5 to 10 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-nv972 18m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-d8qbx 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-bb2r4 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f54779779 Created pod: maas-default-gateway-openshift-default-6f54779779-8l6gm 18m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-nv972 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Container started 18m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-xljwk Container started 18m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Container started 18m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Add eth0 [10.134.1.86/23] from ovn-kubernetes 18m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-xljwk Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-xljwk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-xljwk Add eth0 [10.132.0.232/23] from ovn-kubernetes 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Add eth0 [10.133.0.224/23] from ovn-kubernetes 18m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-nv972 Container started 18m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-nv972 Container created 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Add eth0 [10.132.0.231/23] from ovn-kubernetes 18m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f54779779-nv972 Add eth0 [10.133.0.225/23] from ovn-kubernetes 18m Normal Started pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Container started 18m Normal Created pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Container created 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-nv972 Startup probe failed: Get "http://10.133.0.225:15021/healthz/ready": dial tcp 10.133.0.225:15021: connect: connection refused 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Startup probe failed: Get "http://10.134.1.86:15021/healthz/ready": dial tcp 10.134.1.86:15021: connect: connection refused 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Startup probe failed: Get "http://10.133.0.224:15021/healthz/ready": dial tcp 10.133.0.224:15021: connect: connection refused 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Startup probe error: Get "http://10.134.1.86:15021/healthz/ready": dial tcp 10.134.1.86:15021: connect: connection refused... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Startup probe error: Get "http://10.133.0.224:15021/healthz/ready": dial tcp 10.133.0.224:15021: connect: connection refused... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-nv972 Startup probe error: Get "http://10.133.0.225:15021/healthz/ready": dial tcp 10.133.0.225:15021: connect: connection refused... 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Startup probe failed: Get "http://10.132.0.231:15021/healthz/ready": dial tcp 10.132.0.231:15021: connect: connection refused 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Startup probe error: Get "http://10.132.0.231:15021/healthz/ready": dial tcp 10.132.0.231:15021: connect: connection refused... 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-xljwk Startup probe failed: Get "http://10.132.0.232:15021/healthz/ready": dial tcp 10.132.0.232:15021: connect: connection refused 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-xljwk Startup probe error: Get "http://10.132.0.232:15021/healthz/ready": dial tcp 10.132.0.232:15021: connect: connection refused... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Startup probe error: Get "http://10.133.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Startup probe failed: Get "http://10.133.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Startup probe failed: Get "http://10.134.1.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Startup probe error: Get "http://10.134.1.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-xljwk Startup probe failed: Get "http://10.132.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-nv972 Startup probe failed: Get "http://10.133.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Startup probe failed: Get "http://10.132.0.231:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-nv972 Startup probe error: Get "http://10.133.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-xljwk Startup probe error: Get "http://10.132.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Startup probe error: Get "http://10.132.0.231:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 17m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-54hm4 Stopping container discovery 17m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 17m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-54hm4 16m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Readiness probe failed: Get "http://10.133.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 16m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-d8qbx Readiness probe error: Get "http://10.133.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 16m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Readiness probe failed: Get "http://10.134.1.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 16m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-8l6gm Readiness probe error: Get "http://10.134.1.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 16m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Readiness probe failed: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 16m Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-j8c67 Readiness probe error: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 14m Normal AddedLabel gateway/e2e-worker-single-main-013647 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-013647 14m Normal AddedInterface pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Add eth0 [10.134.1.98/23] from ovn-kubernetes 14m Normal ScalingReplicaSet deployment/e2e-worker-single-main-013647-openshift-default Scaled up replica set e2e-worker-single-main-013647-openshift-default-765765cdb5 from 0 to 1 14m Normal SuccessfulCreate replicaset/e2e-worker-single-main-013647-openshift-default-765765cdb5 Created pod: e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-013647 Scaled up replica set payload-processing-e2e-worker-single-main-013647-6f98d9774b from 0 to 1 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-013647 Scaled up replica set payload-pre-processing-e2e-worker-single-main-013647-5dcfdfc987 from 0 to 1 13m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Container created 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-013647-5dcfdfc987 Error creating: pods "payload-pre-processing-e2e-worker-single-main-013647-5dcfdfc987-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-013647: serviceaccount "payload-processing-e2e-worker-single-main-013647" not found 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Add eth0 [10.134.1.100/23] from ovn-kubernetes 13m Normal Started pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Container started 13m Normal Created pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Container created 13m Normal Started pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Container started 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-013647-5dcfdfc987 Created pod: payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb 13m Normal Pulled pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-013647-6f98d9774b Created pod: payload-processing-e2e-worker-single-main-013647-6f98d977494lzg 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-013647-6f98d9774b Error creating: pods "payload-processing-e2e-worker-single-main-013647-6f98d9774b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-013647: serviceaccount "payload-processing-e2e-worker-single-main-013647" not found 13m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Add eth0 [10.134.1.99/23] from ovn-kubernetes 13m Warning Unhealthy pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Readiness probe failed: Get "http://10.134.1.98:15021/healthz/ready": read tcp 10.134.0.2:47092->10.134.1.98:15021: read: connection reset by peer 13m Warning ProbeError pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Readiness probe error: Get "http://10.134.1.98:15021/healthz/ready": read tcp 10.134.0.2:47092->10.134.1.98:15021: read: connection reset by peer... 13m Normal Pulled pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Container created 13m Normal Started pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Container started 13m Warning ProbeError pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Startup probe error: Get "http://10.134.1.98:15021/healthz/ready": dial tcp 10.134.1.98:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Startup probe failed: Get "http://10.134.1.98:15021/healthz/ready": dial tcp 10.134.1.98:15021: connect: connection refused 13m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-013647-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 13m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-013647-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 13m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-013647-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 13m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-013647-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Normal Killing pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Stopping container payload-processing 12m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Stopping container payload-pre-processing 12m Normal Killing pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs Stopping container istio-proxy 12m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 12m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 10m Normal SuccessfulCreate replicaset/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b5 Created pod: e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t 10m Normal ScalingReplicaSet deployment/e2e-worker-single-main-0bdca7-openshift-default Scaled up replica set e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b5 from 0 to 1 10m Normal AddedLabel gateway/e2e-worker-single-main-0bdca7 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-0bdca7 10m Normal AddedInterface pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Add eth0 [10.134.1.107/23] from ovn-kubernetes 9m52s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-0bdca7 Scaled up replica set payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4b from 0 to 1 9m51s Normal Created pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Container created 9m51s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4b Created pod: payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm 9m51s Normal Pulled pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m51s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-0bdca7 Scaled up replica set payload-pre-processing-e2e-worker-single-main-0bdca7-8489bb8bf9 from 0 to 1 9m51s Normal Started pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Container started 9m51s Normal Started pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Container started 9m51s Normal Created pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Container created 9m51s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m51s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4b Error creating: pods "payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7: serviceaccount "payload-processing-e2e-worker-single-main-0bdca7" not found 9m51s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-0bdca7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m51s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Add eth0 [10.134.1.108/23] from ovn-kubernetes 9m51s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bb8bf9 Created pod: payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh 9m51s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bb8bf9 Error creating: pods "payload-pre-processing-e2e-worker-single-main-0bdca7-8489bb8bf9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7: serviceaccount "payload-processing-e2e-worker-single-main-0bdca7" not found 9m51s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-0bdca7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m51s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Add eth0 [10.134.1.109/23] from ovn-kubernetes 9m38s Normal Created pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Container created 9m38s Normal Started pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Container started 9m38s Normal Pulled pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m32s Warning ProbeError pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Startup probe error: Get "http://10.134.1.107:15021/healthz/ready": dial tcp 10.134.1.107:15021: connect: connection refused... 9m32s Warning Unhealthy pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Startup probe failed: Get "http://10.134.1.107:15021/healthz/ready": dial tcp 10.134.1.107:15021: connect: connection refused 9m6s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-0bdca7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m6s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-0bdca7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m25s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Stopping container payload-pre-processing 8m25s Normal Killing pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Stopping container payload-processing 8m14s Normal Killing pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t Stopping container istio-proxy 8m9s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 8m9s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m15s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-frwdk Readiness probe failed: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m15s Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-frwdk Readiness probe error: Get "http://10.132.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Readiness probe failed: Get "http://10.132.0.231:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m52s Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-xljwk Readiness probe error: Get "http://10.132.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-xljwk Readiness probe failed: Get "http://10.132.0.232:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m52s Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-nv972 Readiness probe error: Get "http://10.133.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f54779779-nv972 Readiness probe failed: Get "http://10.133.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m52s Warning ProbeError pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 Readiness probe error: Get "http://10.132.0.231:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...