=== events in openshift-ingress (2026-10-30T22:54:58Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b to ip-10-0-141-147.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf to ip-10-0-137-81.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh to ip-10-0-141-147.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh to ip-10-0-135-59.us-west-1.compute.internal 39m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b to ip-10-0-135-59.us-west-1.compute.internal 38m Normal Scheduled pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Successfully assigned openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl to ip-10-0-135-59.us-west-1.compute.internal 29m Normal Scheduled pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Successfully assigned openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd to ip-10-0-137-81.us-west-1.compute.internal 28m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v to ip-10-0-137-81.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Successfully assigned openshift-ingress/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt to ip-10-0-141-147.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj to ip-10-0-137-81.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 to ip-10-0-137-81.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 to ip-10-0-141-147.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g to ip-10-0-135-59.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw to ip-10-0-135-59.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb to ip-10-0-135-59.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg to ip-10-0-135-59.us-west-1.compute.internal 56m Normal Scheduled pod/payload-pre-processing-7b98bb678c-f82bg Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-f82bg to ip-10-0-141-147.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm to ip-10-0-141-147.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Successfully assigned openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf to ip-10-0-137-81.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Successfully assigned openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn to ip-10-0-141-147.us-west-1.compute.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Successfully assigned openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz to ip-10-0-135-59.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt to ip-10-0-137-81.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx to ip-10-0-137-81.us-west-1.compute.internal 40m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6dg8m to ip-10-0-141-147.us-west-1.compute.internal 28m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v to ip-10-0-135-59.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Successfully assigned openshift-ingress/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Successfully assigned openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb to ip-10-0-141-147.us-west-1.compute.internal 59m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-72njd to ip-10-0-135-59.us-west-1.compute.internal 33m Normal Scheduled pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc to ip-10-0-135-59.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v to ip-10-0-135-59.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 to ip-10-0-137-81.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff to ip-10-0-137-81.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 to ip-10-0-135-59.us-west-1.compute.internal 28m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs to ip-10-0-141-147.us-west-1.compute.internal 40m Normal Scheduled pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Successfully assigned openshift-ingress/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 to ip-10-0-141-147.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nqjll to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 to ip-10-0-137-81.us-west-1.compute.internal 5m8s Normal Scheduled pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Successfully assigned openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 to ip-10-0-135-59.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt to ip-10-0-135-59.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 to ip-10-0-137-81.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 to ip-10-0-141-147.us-west-1.compute.internal 6m18s Normal Scheduled pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv to ip-10-0-141-147.us-west-1.compute.internal 34m Normal Scheduled pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Successfully assigned openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf to ip-10-0-137-81.us-west-1.compute.internal 53m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw to ip-10-0-141-147.us-west-1.compute.internal 27m Normal Scheduled pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Successfully assigned openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g to ip-10-0-137-81.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Successfully assigned openshift-ingress/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q to ip-10-0-135-59.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mzffc to ip-10-0-141-147.us-west-1.compute.internal 65m Normal Scheduled pod/router-default-d48b6d977-dstqd Successfully assigned openshift-ingress/router-default-d48b6d977-dstqd to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps to ip-10-0-135-59.us-west-1.compute.internal 29m Normal Scheduled pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Successfully assigned openshift-ingress/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz to ip-10-0-141-147.us-west-1.compute.internal 35m Normal Scheduled pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Successfully assigned openshift-ingress/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr to ip-10-0-135-59.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk to ip-10-0-137-81.us-west-1.compute.internal 40m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm to ip-10-0-137-81.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 to ip-10-0-137-81.us-west-1.compute.internal 37m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k to ip-10-0-141-147.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb to ip-10-0-137-81.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm to ip-10-0-137-81.us-west-1.compute.internal 37m Normal Scheduled pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Successfully assigned openshift-ingress/payload-processing-e2e-api-a-512caa-6946957b9-9zspt to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px to ip-10-0-135-59.us-west-1.compute.internal 43m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 to ip-10-0-137-81.us-west-1.compute.internal 6m17s Normal Scheduled pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Successfully assigned openshift-ingress/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr to ip-10-0-141-147.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm to ip-10-0-135-59.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-c2cxq to ip-10-0-141-147.us-west-1.compute.internal 38m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l to ip-10-0-137-81.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Successfully assigned openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m to ip-10-0-141-147.us-west-1.compute.internal 36m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn to ip-10-0-137-81.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 to ip-10-0-141-147.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Successfully assigned openshift-ingress/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 to ip-10-0-141-147.us-west-1.compute.internal 40m Normal Scheduled pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Successfully assigned openshift-ingress/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 to ip-10-0-141-147.us-west-1.compute.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr to ip-10-0-137-81.us-west-1.compute.internal 33m Normal Scheduled pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Successfully assigned openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 to ip-10-0-137-81.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989 to ip-10-0-137-81.us-west-1.compute.internal 61m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-kkskg to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx to ip-10-0-141-147.us-west-1.compute.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 to ip-10-0-137-81.us-west-1.compute.internal 39m Normal Scheduled pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Successfully assigned openshift-ingress/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w to ip-10-0-135-59.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc to ip-10-0-137-81.us-west-1.compute.internal 43m Normal Scheduled pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Successfully assigned openshift-ingress/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb to ip-10-0-137-81.us-west-1.compute.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt to ip-10-0-135-59.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Successfully assigned openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 to ip-10-0-141-147.us-west-1.compute.internal 5m8s Normal Scheduled pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Successfully assigned openshift-ingress/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg to ip-10-0-141-147.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-k2nvr to ip-10-0-141-147.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p to ip-10-0-141-147.us-west-1.compute.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms to ip-10-0-135-59.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 to ip-10-0-135-59.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx to ip-10-0-135-59.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9hjxv to ip-10-0-135-59.us-west-1.compute.internal 39m Normal Scheduled pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Successfully assigned openshift-ingress/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh to ip-10-0-137-81.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Successfully assigned openshift-ingress/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b to ip-10-0-141-147.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf to ip-10-0-135-59.us-west-1.compute.internal 56m Normal Scheduled pod/payload-processing-55fbbbf9dc-wztzx Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-wztzx to ip-10-0-141-147.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8kz9s to ip-10-0-135-59.us-west-1.compute.internal 47m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr to ip-10-0-141-147.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs to ip-10-0-141-147.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d to ip-10-0-141-147.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hdhvg to ip-10-0-135-59.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Successfully assigned openshift-ingress/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd to ip-10-0-141-147.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst to ip-10-0-137-81.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp to ip-10-0-135-59.us-west-1.compute.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 to ip-10-0-135-59.us-west-1.compute.internal 44m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc to ip-10-0-137-81.us-west-1.compute.internal 43m Normal Scheduled pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Successfully assigned openshift-ingress/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h to ip-10-0-135-59.us-west-1.compute.internal 40m Normal Scheduled pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Successfully assigned openshift-ingress/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d to ip-10-0-135-59.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l to ip-10-0-135-59.us-west-1.compute.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Successfully assigned openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 to ip-10-0-137-81.us-west-1.compute.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f to ip-10-0-135-59.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Successfully assigned openshift-ingress/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq to ip-10-0-141-147.us-west-1.compute.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv to ip-10-0-137-81.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Successfully assigned openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 to ip-10-0-141-147.us-west-1.compute.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg to ip-10-0-137-81.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 to ip-10-0-137-81.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7k7bp to ip-10-0-141-147.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px to ip-10-0-141-147.us-west-1.compute.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj to ip-10-0-137-81.us-west-1.compute.internal 62m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-6qbp2 to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p to ip-10-0-137-81.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs to ip-10-0-135-59.us-west-1.compute.internal 6m25s Normal Scheduled pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Successfully assigned openshift-ingress/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz to ip-10-0-141-147.us-west-1.compute.internal 36m Normal Scheduled pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Successfully assigned openshift-ingress/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs to ip-10-0-137-81.us-west-1.compute.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw to ip-10-0-135-59.us-west-1.compute.internal 65m Warning FailedScheduling pod/router-default-d48b6d977-dstqd no nodes available to schedule pods 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rknl4 to ip-10-0-141-147.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dv8z5 to ip-10-0-135-59.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-t54jj to ip-10-0-141-147.us-west-1.compute.internal 34m Normal Scheduled pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Successfully assigned openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr to ip-10-0-137-81.us-west-1.compute.internal 55m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd to ip-10-0-141-147.us-west-1.compute.internal 36m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 to ip-10-0-135-59.us-west-1.compute.internal 55m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk to ip-10-0-141-147.us-west-1.compute.internal 34m Normal Scheduled pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Successfully assigned openshift-ingress/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm to ip-10-0-141-147.us-west-1.compute.internal 5m17s Normal Scheduled pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Successfully assigned openshift-ingress/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Successfully assigned openshift-ingress/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 to ip-10-0-141-147.us-west-1.compute.internal 49m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv to ip-10-0-135-59.us-west-1.compute.internal 51m Normal Scheduled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Successfully assigned openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh to ip-10-0-137-81.us-west-1.compute.internal 54m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq to ip-10-0-141-147.us-west-1.compute.internal 71m Warning FailedScheduling pod/router-default-d48b6d977-dstqd no nodes available to schedule pods 55m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn to ip-10-0-135-59.us-west-1.compute.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd to ip-10-0-135-59.us-west-1.compute.internal 60m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 to ip-10-0-141-147.us-west-1.compute.internal 39m Normal Scheduled pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Successfully assigned openshift-ingress/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc to ip-10-0-135-59.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r to ip-10-0-137-81.us-west-1.compute.internal 49m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f9slk to ip-10-0-135-59.us-west-1.compute.internal 15m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 to ip-10-0-141-147.us-west-1.compute.internal 42m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm to ip-10-0-137-81.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Successfully assigned openshift-ingress/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr to ip-10-0-135-59.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r to ip-10-0-135-59.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 to ip-10-0-135-59.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 to ip-10-0-137-81.us-west-1.compute.internal 54m Normal Scheduled pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv to ip-10-0-135-59.us-west-1.compute.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk to ip-10-0-135-59.us-west-1.compute.internal 61m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs to ip-10-0-141-147.us-west-1.compute.internal 50m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf to ip-10-0-137-81.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-whxxj to ip-10-0-135-59.us-west-1.compute.internal 60m Normal Scheduled pod/kube-auth-proxy-b76569bf4-j5z6p Successfully assigned openshift-ingress/kube-auth-proxy-b76569bf4-j5z6p to ip-10-0-135-59.us-west-1.compute.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z to ip-10-0-135-59.us-west-1.compute.internal 54m Normal Scheduled pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g to ip-10-0-141-147.us-west-1.compute.internal 48m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr to ip-10-0-135-59.us-west-1.compute.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 to ip-10-0-135-59.us-west-1.compute.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw to ip-10-0-137-81.us-west-1.compute.internal 15m Normal Scheduled pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk to ip-10-0-135-59.us-west-1.compute.internal 29m Normal Scheduled pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Successfully assigned openshift-ingress/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc to ip-10-0-141-147.us-west-1.compute.internal 46m Normal Scheduled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Successfully assigned openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x to ip-10-0-135-59.us-west-1.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc to ip-10-0-135-59.us-west-1.compute.internal 45m Normal Scheduled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Successfully assigned openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk to ip-10-0-137-81.us-west-1.compute.internal 54m Normal Scheduled pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 to ip-10-0-141-147.us-west-1.compute.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g to ip-10-0-141-147.us-west-1.compute.internal 55m Normal Scheduled pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh to ip-10-0-141-147.us-west-1.compute.internal 60m Normal Scheduled pod/kube-auth-proxy-b76569bf4-jtdtb Successfully assigned openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb to ip-10-0-137-81.us-west-1.compute.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn to ip-10-0-137-81.us-west-1.compute.internal 35m Normal Scheduled pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Successfully assigned openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z to ip-10-0-137-81.us-west-1.compute.internal 10m Normal Scheduled pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Successfully assigned openshift-ingress/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 to ip-10-0-141-147.us-west-1.compute.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs to ip-10-0-137-81.us-west-1.compute.internal 55m Normal Scheduled pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr to ip-10-0-141-147.us-west-1.compute.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh to ip-10-0-135-59.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws to ip-10-0-137-81.us-west-1.compute.internal 27m Normal Scheduled pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Successfully assigned openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb to ip-10-0-137-81.us-west-1.compute.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml to ip-10-0-137-81.us-west-1.compute.internal 34m Normal Scheduled pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Successfully assigned openshift-ingress/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 to ip-10-0-135-59.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-5l25z to ip-10-0-135-59.us-west-1.compute.internal 56m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vmh57 to ip-10-0-135-59.us-west-1.compute.internal 54m Normal Scheduled pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm to ip-10-0-135-59.us-west-1.compute.internal 55m Normal Scheduled pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc to ip-10-0-141-147.us-west-1.compute.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 to ip-10-0-141-147.us-west-1.compute.internal 15m Normal Scheduled pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Successfully assigned openshift-ingress/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw to ip-10-0-141-147.us-west-1.compute.internal 27m Normal Scheduled pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-2wc6x to ip-10-0-135-59.us-west-1.compute.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 to ip-10-0-141-147.us-west-1.compute.internal 71m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-d48b6d977 from 0 to 1 71m Warning FailedCreate replicaset/router-default-d48b6d977 Error creating: pods "router-default-d48b6d977-" is forbidden: error fetching namespace "openshift-ingress": unable to find annotation openshift.io/sa.scc.uid-range 71m Normal SuccessfulCreate replicaset/router-default-d48b6d977 Created pod: router-default-d48b6d977-dstqd 71m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 71m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 71m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 65m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 65m Warning FailedMount pod/router-default-d48b6d977-dstqd MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 65m Warning FailedMount pod/router-default-d48b6d977-dstqd MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 65m Warning FailedMount pod/router-default-d48b6d977-dstqd MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 65m Warning FailedMount pod/router-default-d48b6d977-dstqd MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 65m Normal AddedInterface pod/router-default-d48b6d977-dstqd Add eth0 [10.132.0.12/23] from ovn-kubernetes 65m Normal Created pod/router-default-d48b6d977-dstqd Container created 65m Normal Pulled pod/router-default-d48b6d977-dstqd Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 65m Normal Started pod/router-default-d48b6d977-dstqd Container started 62m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 62m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Add eth0 [10.133.0.18/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-6qbp2 62m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 62m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 62m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.843s (3.843s including waiting). Image size: 185476477 bytes. 62m Normal Started pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Container started 62m Normal Created pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Container created 62m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Readiness probe failed: HTTP probe failed with statuscode: 503 62m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Readiness probe error: HTTP probe failed with statuscode: 503... 61m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 0 to 1 61m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-kkskg 61m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Add eth0 [10.132.0.35/23] from ovn-kubernetes 61m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-67dd9d5b47 Created pod: openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d 61m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 61m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Add eth0 [10.132.0.34/23] from ovn-kubernetes 61m Normal Pulling pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 61m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 61m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-67dd9d5b47 from 0 to 1 61m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 61m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.403s (9.403s including waiting). Image size: 183555333 bytes. 61m Normal Started pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Container started 61m Normal Created pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Container created 61m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-4665d Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.224s (9.224s including waiting). Image size: 183555333 bytes. 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Normal SuccessfulCreate replicaset/kube-auth-proxy-b76569bf4 Created pod: kube-auth-proxy-b76569bf4-j5z6p 60m Normal SuccessfulCreate replicaset/kube-auth-proxy-b76569bf4 Created pod: kube-auth-proxy-b76569bf4-jtdtb 60m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-b76569bf4 from 0 to 2 60m Normal Pulling pod/kube-auth-proxy-b76569bf4-j5z6p Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 60m Normal AddedInterface pod/kube-auth-proxy-b76569bf4-jtdtb Add eth0 [10.134.0.27/23] from ovn-kubernetes 60m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-6f89d9f6cc from 0 to 1 60m Normal Pulling pod/kube-auth-proxy-b76569bf4-jtdtb Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 60m Normal AddedInterface pod/kube-auth-proxy-b76569bf4-j5z6p Add eth0 [10.133.0.20/23] from ovn-kubernetes 60m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-6f89d9f6cc Created pod: data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 60m Normal Created pod/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 Container created 60m Normal Started pod/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 Container started 60m Normal Pulled pod/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-6f89d9f6cclrhb8 Add eth0 [10.132.0.43/23] from ovn-kubernetes 60m Normal Pulled pod/kube-auth-proxy-b76569bf4-jtdtb Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.752s (2.752s including waiting). Image size: 188904677 bytes. 60m Normal Started pod/kube-auth-proxy-b76569bf4-jtdtb Container started 60m Normal Pulled pod/kube-auth-proxy-b76569bf4-j5z6p Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.716s (2.716s including waiting). Image size: 188904677 bytes. 60m Normal Created pod/kube-auth-proxy-b76569bf4-j5z6p Container created 60m Normal Started pod/kube-auth-proxy-b76569bf4-j5z6p Container started 60m Normal Created pod/kube-auth-proxy-b76569bf4-jtdtb Container created 60m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 59m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-72njd 59m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 59m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Add eth0 [10.133.0.26/23] from ovn-kubernetes 59m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 59m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Container created 59m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Container started 59m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Readiness probe failed: HTTP probe failed with statuscode: 503 59m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-72njd Readiness probe error: HTTP probe failed with statuscode: 503... 59m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Readiness probe failed: Get "http://10.133.0.18:8080/ready": dial tcp 10.133.0.18:8080: connect: connection refused 59m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Readiness probe error: Get "http://10.133.0.18:8080/ready": dial tcp 10.133.0.18:8080: connect: connection refused... 59m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-6qbp2 Stopping container discovery 59m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-6qbp2 59m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 59m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Container started 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Startup probe failed: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Startup probe error: Get "http://10.132.0.34:15021/healthz/ready": dial tcp 10.132.0.34:15021: connect: connection refused... 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-dv8z5 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-ckkdb 56m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 1 to 3 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Add eth0 [10.134.0.38/23] from ovn-kubernetes 56m Normal Pulling pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Add eth0 [10.133.0.31/23] from ovn-kubernetes 56m Normal Pulling pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 56m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Container started 56m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Container created 56m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.923s (3.923s including waiting). Image size: 183555333 bytes. 56m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.813s (3.813s including waiting). Image size: 183555333 bytes. 56m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Container started 56m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Container created 56m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused... 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-vmh57 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-k2nvr 56m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 3 to 6 56m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-zwvws 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Add eth0 [10.133.0.32/23] from ovn-kubernetes 56m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Container created 56m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Container created 56m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Container created 56m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Container started 56m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Container started 56m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Add eth0 [10.132.0.55/23] from ovn-kubernetes 56m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Add eth0 [10.134.0.39/23] from ovn-kubernetes 56m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Container started 56m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Startup probe error: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused... 56m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused... 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Startup probe failed: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused 56m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 56m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 56m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 56m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1 56m Warning FailedCreate replicaset/payload-processing-55fbbbf9dc Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 56m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-f82bg 56m Normal SuccessfulCreate replicaset/payload-processing-55fbbbf9dc Created pod: payload-processing-55fbbbf9dc-wztzx 56m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-f82bg Add eth0 [10.132.0.58/23] from ovn-kubernetes 56m Normal Pulling pod/payload-pre-processing-7b98bb678c-f82bg Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 56m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 56m Normal AddedInterface pod/payload-processing-55fbbbf9dc-wztzx Add eth0 [10.132.0.59/23] from ovn-kubernetes 56m Normal Pulling pod/payload-processing-55fbbbf9dc-wztzx Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 56m Normal Started pod/payload-pre-processing-7b98bb678c-f82bg Container started 56m Normal Created pod/payload-pre-processing-7b98bb678c-f82bg Container created 56m Normal Pulled pod/payload-pre-processing-7b98bb678c-f82bg Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.166s (3.166s including waiting). Image size: 125353965 bytes. 56m Normal Started pod/payload-processing-55fbbbf9dc-wztzx Container started 56m Normal Created pod/payload-processing-55fbbbf9dc-wztzx Container created 56m Normal Pulled pod/payload-processing-55fbbbf9dc-wztzx Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.823s (2.823s including waiting). Image size: 125353965 bytes. 55m Normal AddedLabel gateway/e2e-worker-w1-9cff60 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-9cff60 55m Normal AddedLabel gateway/e2e-worker-w2-d8837a Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-d8837a 55m Normal AddedLabel gateway/e2e-worker-w0-6a0a09 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-6a0a09 55m Normal AddedLabel gateway/e2e-worker-w3-8ca52a Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-8ca52a 55m Normal AddedLabel gateway/e2e-worker-w6-2182a5 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-2182a5 55m Normal AddedLabel gateway/e2e-worker-w5-3783c2 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-3783c2 55m Normal AddedLabel gateway/e2e-worker-w4-bce513 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-bce513 55m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 55m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled up replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 0 to 1 55m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 0 to 1 55m Normal ScalingReplicaSet deployment/e2e-worker-w3-8ca52a-openshift-default Scaled up replica set e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 from 0 to 1 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps 55m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled up replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 0 to 1 55m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Add eth0 [10.133.0.33/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 Created pod: e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz 55m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Add eth0 [10.132.0.61/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Add eth0 [10.132.0.63/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Add eth0 [10.133.0.34/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 0 to 1 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl 55m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Add eth0 [10.132.0.62/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled up replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 0 to 1 55m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 0 to 1 55m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Add eth0 [10.133.0.35/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Add eth0 [10.134.0.40/23] from ovn-kubernetes 55m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-6a0a09 Scaled up replica set payload-processing-e2e-worker-w0-6a0a09-7c97c694d5 from 0 to 1 55m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5 Created pod: payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh 55m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-6a0a09 Scaled up replica set payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9 from 0 to 1 55m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5 Error creating: pods "payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6a0a09: serviceaccount "payload-processing-e2e-worker-w0-6a0a09" not found 55m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9 Created pod: payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b 55m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9 Error creating: pods "payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-6a0a09: serviceaccount "payload-processing-e2e-worker-w0-6a0a09" not found 55m Normal Started pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Container started 55m Normal Created pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Container created 55m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Pulled pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Created pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Container created 55m Normal Started pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Container started 55m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Add eth0 [10.132.0.64/23] from ovn-kubernetes 55m Normal AddedInterface pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Add eth0 [10.132.0.65/23] from ovn-kubernetes 55m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9 Error creating: pods "payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-d8837a: serviceaccount "payload-processing-e2e-worker-w2-d8837a" not found 55m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-d8837a Scaled up replica set payload-pre-processing-e2e-worker-w2-d8837a-67577b7475 from 0 to 1 55m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475 Error creating: pods "payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-d8837a: serviceaccount "payload-processing-e2e-worker-w2-d8837a" not found 55m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-d8837a Scaled up replica set payload-processing-e2e-worker-w2-d8837a-6bdbb559b9 from 0 to 1 55m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9 Created pod: payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc 55m Normal Started pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Container started 55m Normal Created pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Container created 55m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Add eth0 [10.132.0.66/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475 Created pod: payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk 55m Normal AddedInterface pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Add eth0 [10.132.0.67/23] from ovn-kubernetes 55m Normal Started pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Container started 55m Normal Pulled pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Created pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Container created 55m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc Error creating: pods "payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-8ca52a: serviceaccount "payload-processing-e2e-worker-w3-8ca52a" not found 55m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-8ca52a Scaled up replica set payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc from 0 to 1 55m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-8ca52a Scaled up replica set payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff from 0 to 1 55m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff Error creating: pods "payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-8ca52a: serviceaccount "payload-processing-e2e-worker-w3-8ca52a" not found 55m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff Created pod: payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd 55m Normal Created pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Container created 55m Normal AddedInterface pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Add eth0 [10.132.0.69/23] from ovn-kubernetes 55m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal Pulled pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Add eth0 [10.132.0.68/23] from ovn-kubernetes 55m Normal Started pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Container started 55m Normal Created pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Container created 55m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc Created pod: payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr 55m Normal Started pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Container started 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc Error creating: pods "payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-3783c2: serviceaccount "payload-processing-e2e-worker-w5-3783c2" not found 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-3783c2 Scaled up replica set payload-processing-e2e-worker-w5-3783c2-846f9695b9 from 0 to 1 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc Created pod: payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-3783c2-846f9695b9 Created pod: payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-3783c2 Scaled up replica set payload-pre-processing-e2e-worker-w5-3783c2-55645569bc from 0 to 1 54m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-3783c2-846f9695b9 Error creating: pods "payload-processing-e2e-worker-w5-3783c2-846f9695b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-3783c2: serviceaccount "payload-processing-e2e-worker-w5-3783c2" not found 54m Normal Started pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Container started 54m Normal AddedInterface pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Add eth0 [10.132.0.70/23] from ovn-kubernetes 54m Normal Pulling pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 54m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Add eth0 [10.134.0.42/23] from ovn-kubernetes 54m Normal Pulled pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Container created 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-2182a5 Scaled up replica set payload-processing-e2e-worker-w6-2182a5-597f4f4f45 from 0 to 1 54m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Add eth0 [10.133.0.38/23] from ovn-kubernetes 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-2182a5 Scaled up replica set payload-pre-processing-e2e-worker-w6-2182a5-97598c48c from 0 to 1 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c Error creating: pods "payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-2182a5: serviceaccount "payload-processing-e2e-worker-w6-2182a5" not found 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-2182a5-597f4f4f45 Created pod: payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c Created pod: payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f 54m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-2182a5-597f4f4f45 Error creating: pods "payload-processing-e2e-worker-w6-2182a5-597f4f4f45-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-2182a5: serviceaccount "payload-processing-e2e-worker-w6-2182a5" not found 54m Normal Pulling pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 54m Normal Started pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Container started 54m Normal Pulling pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 54m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.943s (2.943s including waiting). Image size: 125353965 bytes. 54m Normal Created pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Container created 54m Normal AddedInterface pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Add eth0 [10.133.0.39/23] from ovn-kubernetes 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-bce513 Scaled up replica set payload-processing-e2e-worker-w4-bce513-556ddf7445 from 0 to 1 54m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-bce513-556ddf7445 Error creating: pods "payload-processing-e2e-worker-w4-bce513-556ddf7445-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-bce513: serviceaccount "payload-processing-e2e-worker-w4-bce513" not found 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-bce513 Scaled up replica set payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff from 0 to 1 54m Normal AddedInterface pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Add eth0 [10.132.0.73/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff Created pod: payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff Error creating: pods "payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-bce513: serviceaccount "payload-processing-e2e-worker-w4-bce513" not found 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-bce513-556ddf7445 Created pod: payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 54m Normal Pulled pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Container created 54m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Add eth0 [10.133.0.40/23] from ovn-kubernetes 54m Normal Pulling pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 54m Normal Started pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Container started 54m Normal Created pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Container created 54m Normal Created pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Container created 54m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-9cff60-796b546677 Error creating: pods "payload-processing-e2e-worker-w1-9cff60-796b546677-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-9cff60: serviceaccount "payload-processing-e2e-worker-w1-9cff60" not found 54m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c Created pod: payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq 54m Normal Pulled pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 4.552s (4.552s including waiting). Image size: 125353965 bytes. 54m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-9cff60 Scaled up replica set payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c from 0 to 1 54m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c Error creating: pods "payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-9cff60: serviceaccount "payload-processing-e2e-worker-w1-9cff60" not found 54m Normal Started pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Container started 54m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 4.687s (4.687s including waiting). Image size: 125353965 bytes. 54m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-9cff60 Scaled up replica set payload-processing-e2e-worker-w1-9cff60-796b546677 from 0 to 1 54m Normal Started pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Container started 54m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-9cff60-796b546677 Created pod: payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm 54m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Add eth0 [10.132.0.74/23] from ovn-kubernetes 54m Normal Started pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Container started 54m Normal AddedInterface pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Add eth0 [10.133.0.41/23] from ovn-kubernetes 54m Normal Started pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Container started 54m Normal Pulled pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Container created 54m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 54m Normal Created pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Container created 54m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.396s (2.396s including waiting). Image size: 125353965 bytes. 54m Normal Started pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Container started 54m Normal Created pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Container created 54m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Container started 54m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Container created 54m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Startup probe error: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused... 54m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Container created 54m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Container started 54m Normal Created pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Container created 54m Normal Pulled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Container started 54m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Startup probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Startup probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning Unhealthy pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused... 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Container started 54m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Container created 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused... 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 54m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Container started 54m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Container created 54m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Startup probe failed: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Startup probe error: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused... 54m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Container created 54m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Container started 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 54m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused... 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Container started 53m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Container created 53m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused... 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 1 to 2 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw 53m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Container created 53m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Container started 53m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Add eth0 [10.132.0.76/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 53m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 53m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Container started 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 2 to 3 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 51m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Container created 51m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Add eth0 [10.134.0.50/23] from ovn-kubernetes 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh 51m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf 50m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled up replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 1 to 2 50m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Container created 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 50m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Container started 50m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Add eth0 [10.134.0.51/23] from ovn-kubernetes 50m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-ckkdb 50m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 6 to 4 50m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-dv8z5 Stopping container istio-proxy 50m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb Stopping container istio-proxy 50m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-dv8z5 50m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Startup probe failed: Get "http://10.134.0.51:15021/healthz/ready": dial tcp 10.134.0.51:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Startup probe error: Get "http://10.134.0.51:15021/healthz/ready": dial tcp 10.134.0.51:15021: connect: connection refused... 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 1 to 2 50m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Add eth0 [10.133.0.49/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Container started 50m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Container created 50m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled up replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 1 to 2 50m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Add eth0 [10.134.0.52/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Container started 50m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Container created 50m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Startup probe failed: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Startup probe error: Get "http://10.134.0.52:15021/healthz/ready": dial tcp 10.134.0.52:15021: connect: connection refused... 50m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled up replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 1 to 2 50m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Add eth0 [10.132.0.88/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj 50m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Container created 50m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Container started 50m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj 49m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 2 to 3 49m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Container created 49m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Container started 49m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Add eth0 [10.134.0.53/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-hdhvg 49m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 4 to 8 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-9q989 49m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 Created pod: maas-default-gateway-openshift-default-67bf986f99-7k7bp 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Container started 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Add eth0 [10.134.0.54/23] from ovn-kubernetes 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Container created 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Container created 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Add eth0 [10.133.0.51/23] from ovn-kubernetes 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Container created 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Container started 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Add eth0 [10.132.0.92/23] from ovn-kubernetes 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Add eth0 [10.133.0.50/23] from ovn-kubernetes 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Container started 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Container created 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Container started 49m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Startup probe failed: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Startup probe error: Get "http://10.134.0.53:15021/healthz/ready": dial tcp 10.134.0.53:15021: connect: connection refused... 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Startup probe error: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Startup probe failed: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Startup probe error: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Startup probe failed: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Startup probe failed: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Startup probe error: Get "http://10.133.0.51:15021/healthz/ready": dial tcp 10.133.0.51:15021: connect: connection refused... 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 49m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: All metrics below target 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Container created 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Container started 49m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Container created 49m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Add eth0 [10.132.0.97/23] from ovn-kubernetes 49m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Add eth0 [10.134.0.55/23] from ovn-kubernetes 49m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Container started 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Startup probe failed: Get "http://10.134.0.55:15021/healthz/ready": dial tcp 10.134.0.55:15021: connect: connection refused 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Startup probe error: Get "http://10.134.0.55:15021/healthz/ready": dial tcp 10.134.0.55:15021: connect: connection refused... 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 49m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Startup probe failed: Get "http://10.134.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal SuccessfulCreate replicaset/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 Created pod: e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn 49m Normal ScalingReplicaSet deployment/e2e-worker-w3-8ca52a-openshift-default Scaled up replica set e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 from 1 to 2 49m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Startup probe error: Get "http://10.134.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Normal Pulled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Add eth0 [10.132.0.102/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Container started 49m Normal Created pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Container created 49m Warning Unhealthy pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Startup probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Startup probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 49m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Add eth0 [10.132.0.104/23] from ovn-kubernetes 49m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Add eth0 [10.133.0.54/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 49m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 3 to 5 49m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Container created 49m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Container started 49m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Container created 49m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Container started 49m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Startup probe error: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Startup probe failed: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused 49m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Startup probe failed: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Startup probe error: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused... 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 48m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled up replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 2 to 3 48m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Container started 48m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled up replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 2 to 4 48m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Container created 48m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Add eth0 [10.133.0.55/23] from ovn-kubernetes 48m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn 48m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp 48m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 48m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Container created 48m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm 48m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Add eth0 [10.132.0.106/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled up replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 2 to 4 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 48m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Container created 48m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Container started 48m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Add eth0 [10.133.0.56/23] from ovn-kubernetes 48m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 48m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Container started 48m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Container created 48m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Container started 48m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Container started 48m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Container created 48m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Add eth0 [10.134.0.57/23] from ovn-kubernetes 48m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Add eth0 [10.134.0.56/23] from ovn-kubernetes 48m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Startup probe failed: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused 48m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Startup probe error: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Startup probe error: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Startup probe failed: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused 48m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Startup probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Startup probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 48m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 48m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 5 to 7 48m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 48m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr 48m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 48m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled up replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 3 to 5 48m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Add eth0 [10.133.0.60/23] from ovn-kubernetes 48m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Add eth0 [10.133.0.59/23] from ovn-kubernetes 48m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Container started 48m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Container created 48m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled up replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 4 to 6 48m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Add eth0 [10.134.0.61/23] from ovn-kubernetes 48m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 48m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Container created 48m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Container created 48m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Add eth0 [10.133.0.58/23] from ovn-kubernetes 48m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Container started 48m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Container started 48m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Container started 48m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Container created 48m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Container started 48m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Add eth0 [10.134.0.62/23] from ovn-kubernetes 48m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Container created 48m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Add eth0 [10.132.0.108/23] from ovn-kubernetes 48m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r 48m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs 48m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Container started 48m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Container created 48m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Startup probe error: Get "http://10.132.0.108:15021/healthz/ready": dial tcp 10.132.0.108:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Startup probe failed: Get "http://10.132.0.108:15021/healthz/ready": dial tcp 10.132.0.108:15021: connect: connection refused 48m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 48m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused 48m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 48m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 48m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 47m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 3 to 4 47m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 1 to 2 47m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Add eth0 [10.132.0.111/23] from ovn-kubernetes 47m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 47m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 47m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Container created 47m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Add eth0 [10.132.0.112/23] from ovn-kubernetes 47m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Container started 47m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Container created 47m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Container started 47m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Startup probe failed: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused 47m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Startup probe error: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Startup probe failed: Get "http://10.132.0.112:15021/healthz/ready": dial tcp 10.132.0.112:15021: connect: connection refused 47m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Startup probe error: Get "http://10.132.0.112:15021/healthz/ready": dial tcp 10.132.0.112:15021: connect: connection refused... 46m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled up replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 4 to 7 46m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 46m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 46m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs 46m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Container started 46m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Add eth0 [10.132.0.114/23] from ovn-kubernetes 46m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Container started 46m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Container created 46m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Add eth0 [10.132.0.113/23] from ovn-kubernetes 46m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Container created 46m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Container created 46m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Add eth0 [10.133.0.62/23] from ovn-kubernetes 46m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Container started 46m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Startup probe error: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Startup probe failed: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Startup probe failed: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused 46m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 46m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Startup probe error: Get "http://10.132.0.113:15021/healthz/ready": dial tcp 10.132.0.113:15021: connect: connection refused... 46m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Add eth0 [10.134.0.65/23] from ovn-kubernetes 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 46m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Add eth0 [10.133.0.63/23] from ovn-kubernetes 46m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled up replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 5 to 10 46m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Add eth0 [10.132.0.115/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 (combined from similar events): Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 46m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px 46m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Add eth0 [10.134.0.64/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x 46m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r 46m Normal SuccessfulCreate replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Created pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 46m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Container created 46m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Container started 46m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Container created 46m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Container created 46m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Container started 46m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Container created 46m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Container started 46m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Container started 46m Normal Pulled pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Container started 46m Normal AddedInterface pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Add eth0 [10.133.0.64/23] from ovn-kubernetes 46m Normal Created pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Container created 46m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Startup probe error: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Startup probe failed: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Startup probe failed: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Startup probe failed: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused 46m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 46m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Startup probe error: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused... 46m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Startup probe error: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused... 46m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 45m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled up replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 6 to 10 45m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Add eth0 [10.134.0.67/23] from ovn-kubernetes 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 45m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk 45m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs 45m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg 45m Normal SuccessfulCreate replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc (combined from similar events): Created pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 45m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Add eth0 [10.132.0.119/23] from ovn-kubernetes 45m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Container created 45m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Container created 45m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled up replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 7 to 10 45m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 (combined from similar events): Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 45m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Container created 45m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Container started 45m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Container started 45m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Add eth0 [10.133.0.67/23] from ovn-kubernetes 45m Normal Created pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Container created 45m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Add eth0 [10.134.0.69/23] from ovn-kubernetes 45m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Add eth0 [10.134.0.68/23] from ovn-kubernetes 45m Normal AddedInterface pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Add eth0 [10.132.0.120/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh 45m Normal SuccessfulCreate replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Created pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 45m Normal AddedInterface pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Add eth0 [10.133.0.66/23] from ovn-kubernetes 45m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Container started 45m Normal Started pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Container started 45m Normal Pulled pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Container started 45m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Container started 45m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Container created 45m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Container started 45m Normal Pulled pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Container created 45m Normal Created pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Container created 45m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 45m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Startup probe failed: Get "http://10.132.0.119:15021/healthz/ready": dial tcp 10.132.0.119:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Startup probe error: Get "http://10.132.0.119:15021/healthz/ready": dial tcp 10.132.0.119:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Startup probe failed: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Startup probe error: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Startup probe error: Get "http://10.132.0.120:15021/healthz/ready": dial tcp 10.132.0.120:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Startup probe failed: Get "http://10.132.0.120:15021/healthz/ready": dial tcp 10.132.0.120:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Startup probe error: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Startup probe failed: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 45m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Add eth0 [10.132.0.125/23] from ovn-kubernetes 45m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled up replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 7 to 10 45m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Add eth0 [10.134.0.70/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh 45m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 45m Normal AddedInterface pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Add eth0 [10.133.0.69/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 (combined from similar events): Created pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px 45m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 4 to 8 45m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Container started 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 45m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Add eth0 [10.134.0.71/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 45m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Container started 45m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Container created 45m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw 45m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb 45m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg 45m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l 45m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 2 to 4 45m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Add eth0 [10.132.0.126/23] from ovn-kubernetes 45m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Add eth0 [10.133.0.70/23] from ovn-kubernetes 45m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Pulled pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Container created 45m Normal Started pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Container started 45m Normal Created pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Container created 45m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Container created 45m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Add eth0 [10.133.0.71/23] from ovn-kubernetes 45m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Container created 45m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Container created 45m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Add eth0 [10.132.0.127/23] from ovn-kubernetes 45m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Container created 45m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Container started 45m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Container started 45m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Container started 45m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Container started 45m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Add eth0 [10.133.0.72/23] from ovn-kubernetes 45m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Container started 45m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Container created 45m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Container started 45m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Container created 45m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Startup probe error: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Startup probe failed: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused 45m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Startup probe error: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Startup probe failed: Get "http://10.132.0.125:15021/healthz/ready": dial tcp 10.132.0.125:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Startup probe error: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Startup probe failed: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Startup probe error: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Startup probe failed: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Startup probe error: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Startup probe failed: Get "http://10.132.0.126:15021/healthz/ready": dial tcp 10.132.0.126:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Startup probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Startup probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused 45m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Startup probe failed: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused 45m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Startup probe error: Get "http://10.132.0.127:15021/healthz/ready": dial tcp 10.132.0.127:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused 44m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx 44m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Add eth0 [10.134.0.72/23] from ovn-kubernetes 44m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled up replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 8 to 10 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 10; reason: All metrics below target 44m Normal SuccessfulCreate replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 (combined from similar events): Created pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm 44m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Container started 44m Normal AddedInterface pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Add eth0 [10.132.0.128/23] from ovn-kubernetes 44m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Container created 44m Normal Created pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Container created 44m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Pulled pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Container started 44m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 10 to 7 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-rknl4 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-k2nvr 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-7k7bp 44m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-rknl4 Stopping container istio-proxy 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Stopping container istio-proxy 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Stopping container istio-proxy 44m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-k2nvr Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Startup probe error: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Startup probe failed: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused 44m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Readiness probe error: Get "http://10.132.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-7k7bp Readiness probe failed: Get "http://10.132.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal ScalingReplicaSet deployment/e2e-worker-w3-8ca52a-openshift-default Scaled up replica set e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 from 2 to 3 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 44m Normal AddedInterface pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Add eth0 [10.134.0.73/23] from ovn-kubernetes 44m Normal Pulled pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Container created 44m Normal Started pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Container started 44m Normal SuccessfulCreate replicaset/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 Created pod: e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-9q989 44m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 7 to 5 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-hdhvg 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Stopping container istio-proxy 44m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-9q989 Stopping container istio-proxy 44m Warning ProbeError pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Readiness probe error: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Readiness probe failed: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Readiness probe error: Get "http://10.133.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-hdhvg Readiness probe failed: Get "http://10.133.0.50:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-f9slk 44m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-67bf986f99 from 5 to 4 44m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-f9slk Stopping container istio-proxy 44m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 44m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc 44m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 4 to 8 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 44m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx 44m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Add eth0 [10.134.0.74/23] from ovn-kubernetes 44m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Add eth0 [10.133.0.73/23] from ovn-kubernetes 44m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Add eth0 [10.132.0.143/23] from ovn-kubernetes 44m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Add eth0 [10.133.0.74/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d 44m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Container created 44m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Container created 44m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Container started 44m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Container created 44m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Container started 44m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Container started 44m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Container started 44m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Container created 44m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Startup probe failed: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Startup probe error: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Startup probe failed: Get "http://10.132.0.143:15021/healthz/ready": dial tcp 10.132.0.143:15021: connect: connection refused 44m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Startup probe error: Get "http://10.132.0.143:15021/healthz/ready": dial tcp 10.132.0.143:15021: connect: connection refused... 44m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 8 to 9 43m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 9; reason: All metrics below target 43m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Add eth0 [10.134.0.75/23] from ovn-kubernetes 43m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Container created 43m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Readiness probe error: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Readiness probe failed: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Container started 43m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Readiness probe error: Get "http://10.132.0.111:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Readiness probe failed: Get "http://10.132.0.111:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Startup probe error: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Startup probe failed: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused 43m Normal ScalingReplicaSet deployment/e2e-migrate-4cce86f1-gw-openshift-default Scaled up replica set e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9 from 0 to 1 43m Normal AddedLabel gateway/e2e-migrate-4cce86f1-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-4cce86f1-gw 43m Normal SuccessfulCreate replicaset/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9 Created pod: e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb 43m Normal AddedInterface pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Add eth0 [10.132.0.156/23] from ovn-kubernetes 43m Normal AddedLabel gateway/e2e-ait-08235cc9 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-08235cc9 43m Normal ScalingReplicaSet deployment/e2e-ait-08235cc9-openshift-default Scaled up replica set e2e-ait-08235cc9-openshift-default-5656979cc5 from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-ait-08235cc9-openshift-default-5656979cc5 Created pod: e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt 43m Normal AddedInterface pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Add eth0 [10.132.0.157/23] from ovn-kubernetes 43m Normal AddedLabel gateway/e2e-ait-643a82de Added label istio.io/rev=openshift-gateway to gateway e2e-ait-643a82de 43m Normal ScalingReplicaSet deployment/e2e-ait-643a82de-openshift-default Scaled up replica set e2e-ait-643a82de-openshift-default-58f6b7b6b7 from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-ait-643a82de-openshift-default-58f6b7b6b7 Created pod: e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h 43m Normal AddedInterface pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Add eth0 [10.133.0.75/23] from ovn-kubernetes 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5 Error creating: pods "payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-4cce86f1: serviceaccount "payload-processing-e2e-migrate-4cce86f1" not found 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-4cce86f1-fc5594977 Created pod: payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-4cce86f1 Scaled up replica set payload-processing-e2e-migrate-4cce86f1-fc5594977 from 0 to 1 43m Warning FailedCreate replicaset/payload-processing-e2e-migrate-4cce86f1-fc5594977 Error creating: pods "payload-processing-e2e-migrate-4cce86f1-fc5594977-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-4cce86f1: serviceaccount "payload-processing-e2e-migrate-4cce86f1" not found 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-4cce86f1 Scaled up replica set payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5 Created pod: payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf 43m Normal Created pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Container created 43m Normal AddedInterface pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Add eth0 [10.132.0.159/23] from ovn-kubernetes 43m Normal Pulled pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Add eth0 [10.132.0.158/23] from ovn-kubernetes 43m Normal Pulled pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Started pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Container started 43m Normal Created pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Container created 43m Normal Started pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Container started 43m Normal SuccessfulCreate replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 (combined from similar events): Created pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled up replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 9 to 10 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-08235cc9 Scaled up replica set payload-pre-processing-e2e-ait-08235cc9-8d658d675 from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-08235cc9 Scaled up replica set payload-processing-e2e-ait-08235cc9-67f47cc97 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-08235cc9-8d658d675 Created pod: payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g 43m Warning FailedCreate replicaset/payload-processing-e2e-ait-08235cc9-67f47cc97 Error creating: pods "payload-processing-e2e-ait-08235cc9-67f47cc97-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-08235cc9: serviceaccount "payload-processing-e2e-ait-08235cc9" not found 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-08235cc9-8d658d675 Error creating: pods "payload-pre-processing-e2e-ait-08235cc9-8d658d675-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-08235cc9: serviceaccount "payload-processing-e2e-ait-08235cc9" not found 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-08235cc9-67f47cc97 Created pod: payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 43m Normal AddedInterface pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Add eth0 [10.132.0.160/23] from ovn-kubernetes 43m Normal Pulled pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Created pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Container created 43m Normal Started pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Container started 43m Normal AddedInterface pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Add eth0 [10.133.0.77/23] from ovn-kubernetes 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Readiness probe error: Get "http://10.133.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-643a82de Scaled up replica set payload-pre-processing-e2e-ait-643a82de-bc8b4755 from 0 to 1 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Readiness probe failed: Get "http://10.134.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Normal AddedInterface pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Add eth0 [10.133.0.76/23] from ovn-kubernetes 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Readiness probe failed: Get "http://10.133.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Readiness probe error: Get "http://10.134.0.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-643a82de Scaled up replica set payload-processing-e2e-ait-643a82de-6bc94cdfb9 from 0 to 1 43m Normal Pulled pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Readiness probe error: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Readiness probe failed: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning FailedCreate replicaset/payload-processing-e2e-ait-643a82de-6bc94cdfb9 Error creating: pods "payload-processing-e2e-ait-643a82de-6bc94cdfb9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-643a82de: serviceaccount "payload-processing-e2e-ait-643a82de" not found 43m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Readiness probe error: Get "http://10.134.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Readiness probe failed: Get "http://10.132.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-643a82de-6bc94cdfb9 Created pod: payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 43m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Readiness probe failed: Get "http://10.133.0.35:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Readiness probe error: Get "http://10.132.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-643a82de-bc8b4755 Created pod: payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-643a82de-bc8b4755 Error creating: pods "payload-pre-processing-e2e-ait-643a82de-bc8b4755-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-643a82de: serviceaccount "payload-processing-e2e-ait-643a82de" not found 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Readiness probe failed: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Startup probe error: Get "http://10.132.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Readiness probe failed: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Readiness probe failed: Get "http://10.133.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Readiness probe error: Get "http://10.133.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Readiness probe error: Get "http://10.133.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Startup probe failed: Get "http://10.132.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Readiness probe error: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Readiness probe failed: Get "http://10.133.0.60:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Readiness probe error: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Readiness probe failed: Get "http://10.133.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Readiness probe error: Get "http://10.133.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal Pulled pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Created pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Container created 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Readiness probe failed: Get "http://10.132.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Readiness probe failed: Get "http://10.132.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Readiness probe failed: Get "http://10.133.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Readiness probe error: Get "http://10.133.0.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Readiness probe error: Get "http://10.132.0.108:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Readiness probe error: Get "http://10.132.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal Started pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Container started 43m Normal Created pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Container created 43m Normal Started pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Container started 43m Normal AddedInterface pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Add eth0 [10.133.0.78/23] from ovn-kubernetes 43m Normal Pulled pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Pulled pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Add eth0 [10.132.0.162/23] from ovn-kubernetes 43m Normal Created pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Container created 43m Normal Started pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Container started 43m Normal Created pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Container created 43m Normal Started pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Container started 43m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Add eth0 [10.134.0.76/23] from ovn-kubernetes 43m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 43m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-pgvh7 43m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 43m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Startup probe failed: Get "http://10.132.0.160:15021/healthz/ready": dial tcp 10.132.0.160:15021: connect: connection refused 43m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 43m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Startup probe error: Get "http://10.132.0.160:15021/healthz/ready": dial tcp 10.132.0.160:15021: connect: connection refused... 43m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.014s (4.014s including waiting). Image size: 185476477 bytes. 43m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Container created 43m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Container started 43m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Readiness probe error: HTTP probe failed with statuscode: 503... 43m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Readiness probe failed: HTTP probe failed with statuscode: 503 43m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 43m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 Deleted pod: maas-default-gateway-openshift-default-67bf986f99-plwzc 43m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-plwzc Stopping container istio-proxy 42m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Add eth0 [10.132.0.165/23] from ovn-kubernetes 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Add eth0 [10.134.0.77/23] from ovn-kubernetes 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Container created 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Container started 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Add eth0 [10.133.0.81/23] from ovn-kubernetes 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Container started 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Container created 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Container created 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Container started 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 42m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Startup probe failed: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused 42m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 42m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Startup probe error: Get "http://10.132.0.165:15021/healthz/ready": dial tcp 10.132.0.165:15021: connect: connection refused... 42m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Container created 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Add eth0 [10.134.0.78/23] from ovn-kubernetes 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Container started 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Add eth0 [10.132.0.167/23] from ovn-kubernetes 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Container created 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Container started 42m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Startup probe error: Get "http://10.132.0.167:15021/healthz/ready": dial tcp 10.132.0.167:15021: connect: connection refused... 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused 42m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused... 42m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Startup probe failed: Get "http://10.132.0.167:15021/healthz/ready": dial tcp 10.132.0.167:15021: connect: connection refused 42m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-67bf986f99 (combined from similar events): Created pod: maas-default-gateway-openshift-default-67bf986f99-mzffc 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Add eth0 [10.132.0.168/23] from ovn-kubernetes 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Container created 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Container started 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Container started 42m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Container created 42m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Add eth0 [10.133.0.83/23] from ovn-kubernetes 41m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Startup probe error: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused... 41m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Startup probe failed: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused 41m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Startup probe error: Get "http://10.132.0.168:15021/healthz/ready": dial tcp 10.132.0.168:15021: connect: connection refused... 41m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Startup probe failed: Get "http://10.132.0.168:15021/healthz/ready": dial tcp 10.132.0.168:15021: connect: connection refused 41m Normal Started pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Container started 41m Normal Pulled pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Container created 41m Normal Pulled pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Container created 41m Normal Started pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Container started 41m Warning Unhealthy pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Startup probe failed: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused 41m Warning ProbeError pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Startup probe error: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused... 41m Normal Killing pod/payload-processing-e2e-migrate-4cce86f1-fc5594977-9w245 Stopping container payload-processing 41m Warning Unhealthy pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Startup probe failed: Get "http://10.132.0.157:15021/healthz/ready": dial tcp 10.132.0.157:15021: connect: connection refused 41m Warning ProbeError pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Startup probe error: Get "http://10.132.0.157:15021/healthz/ready": dial tcp 10.132.0.157:15021: connect: connection refused... 41m Normal Killing pod/payload-pre-processing-e2e-migrate-4cce86f1-5885475bc5-286hf Stopping container payload-pre-processing 41m Warning ProbeError pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Readiness probe error: Get "http://10.132.0.156:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-08235cc9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-08235cc9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning Unhealthy pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Readiness probe failed: Get "http://10.132.0.156:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Normal Killing pod/payload-processing-e2e-ait-08235cc9-67f47cc97-zf7d7 Stopping container payload-processing 41m Normal Killing pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Stopping container payload-pre-processing 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-643a82de-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-643a82de-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning Unhealthy pod/payload-pre-processing-e2e-ait-08235cc9-8d658d675-mvn9g Readiness probe failed: timeout: failed to connect service "10.133.0.76:50052" within 1s: context deadline exceeded 41m Normal Pulled pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Container created 41m Normal Started pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Container started 41m Normal Killing pod/payload-processing-e2e-ait-643a82de-6bc94cdfb9-8vt55 Stopping container payload-processing 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-4cce86f1-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-4cce86f1-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Normal Killing pod/payload-pre-processing-e2e-ait-643a82de-bc8b4755-5wzt7 Stopping container payload-pre-processing 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-08235cc9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-08235cc9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning Unhealthy pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Startup probe failed: Get "http://10.132.0.156:15021/healthz/ready": dial tcp 10.132.0.156:15021: connect: connection refused 41m Warning ProbeError pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Startup probe error: Get "http://10.132.0.156:15021/healthz/ready": dial tcp 10.132.0.156:15021: connect: connection refused... 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-643a82de-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-643a82de-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-4cce86f1-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-4cce86f1-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal Killing pod/e2e-migrate-4cce86f1-gw-openshift-default-6bdcf6cfd9-6rtmb Stopping container istio-proxy 40m Normal AddedLabel gateway/e2e-ait-3ef8140e Added label istio.io/rev=openshift-gateway to gateway e2e-ait-3ef8140e 40m Normal ScalingReplicaSet deployment/e2e-ait-3ef8140e-openshift-default Scaled up replica set e2e-ait-3ef8140e-openshift-default-6df8574985 from 0 to 1 40m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk 40m Normal Killing pod/e2e-ait-08235cc9-openshift-default-5656979cc5-4rcnt Stopping container istio-proxy 40m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Add eth0 [10.132.0.173/23] from ovn-kubernetes 40m Normal AddedLabel gateway/e2e-derive-781ccf27 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-781ccf27 40m Normal ScalingReplicaSet deployment/e2e-derive-781ccf27-openshift-default Scaled up replica set e2e-derive-781ccf27-openshift-default-79c47976f from 0 to 1 40m Normal AddedInterface pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Add eth0 [10.133.0.84/23] from ovn-kubernetes 40m Normal SuccessfulCreate replicaset/e2e-derive-781ccf27-openshift-default-79c47976f Created pod: e2e-derive-781ccf27-openshift-default-79c47976f-24n2d 40m Warning FailedCreate replicaset/payload-processing-e2e-ait-3ef8140e-bc85668dc Error creating: pods "payload-processing-e2e-ait-3ef8140e-bc85668dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-3ef8140e: serviceaccount "payload-processing-e2e-ait-3ef8140e" not found 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-3ef8140e Scaled up replica set payload-pre-processing-e2e-ait-3ef8140e-87bbbff55 from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-3ef8140e Scaled up replica set payload-processing-e2e-ait-3ef8140e-bc85668dc from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-3ef8140e-bc85668dc Created pod: payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55 Error creating: pods "payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-3ef8140e: serviceaccount "payload-processing-e2e-ait-3ef8140e" not found 40m Normal Killing pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Stopping container istio-proxy 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55 Created pod: payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv 40m Normal Started pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Container started 40m Normal AddedInterface pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Add eth0 [10.132.0.174/23] from ovn-kubernetes 40m Warning Unhealthy pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal AddedInterface pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Add eth0 [10.132.0.175/23] from ovn-kubernetes 40m Normal Started pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Container started 40m Warning ProbeError pod/e2e-ait-643a82de-openshift-default-58f6b7b6b7-8f84h Readiness probe error: HTTP probe failed with statuscode: 503... 40m Normal Created pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Container created 40m Normal Pulled pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Container created 40m Normal Pulled pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-781ccf27 Scaled up replica set payload-processing-e2e-derive-781ccf27-8698b866fc from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-781ccf27 Scaled up replica set payload-pre-processing-e2e-derive-781ccf27-675589c8f9 from 0 to 1 40m Warning FailedCreate replicaset/payload-processing-e2e-derive-781ccf27-8698b866fc Error creating: pods "payload-processing-e2e-derive-781ccf27-8698b866fc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-781ccf27: serviceaccount "payload-processing-e2e-derive-781ccf27" not found 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-781ccf27-8698b866fc Created pod: payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-781ccf27-675589c8f9 Error creating: pods "payload-pre-processing-e2e-derive-781ccf27-675589c8f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-781ccf27: serviceaccount "payload-processing-e2e-derive-781ccf27" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-781ccf27-675589c8f9 Created pod: payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms 40m Normal AddedInterface pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Add eth0 [10.133.0.85/23] from ovn-kubernetes 40m Normal Started pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Container started 40m Normal Created pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Container created 40m Normal AddedInterface pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Add eth0 [10.132.0.176/23] from ovn-kubernetes 40m Normal Pulled pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Container created 40m Normal Started pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Container started 40m Normal Pulled pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 40m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 40m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-6dg8m 40m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Add eth0 [10.132.0.177/23] from ovn-kubernetes 40m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 40m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 6; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled down replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 10 to 6 40m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-bgtps 40m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-458px Stopping container istio-proxy 40m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-7bpsv Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Container started 40m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Container created 40m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.593s (4.593s including waiting). Image size: 185476477 bytes. 40m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-ns784 Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Readiness probe error: HTTP probe failed with statuscode: 503... 39m Normal AddedLabel gateway/e2e-mt-7422b4b3 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-7422b4b3 39m Normal SuccessfulCreate replicaset/e2e-mt-7422b4b3-openshift-default-69d88b745c Created pod: e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w 39m Normal ScalingReplicaSet deployment/e2e-mt-7422b4b3-openshift-default Scaled up replica set e2e-mt-7422b4b3-openshift-default-69d88b745c from 0 to 1 39m Normal AddedInterface pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Add eth0 [10.133.0.90/23] from ovn-kubernetes 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-7422b4b3 Scaled up replica set payload-processing-e2e-mt-7422b4b3-7457ddb899 from 0 to 1 39m Warning FailedCreate replicaset/payload-processing-e2e-mt-7422b4b3-7457ddb899 Error creating: pods "payload-processing-e2e-mt-7422b4b3-7457ddb899-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-7422b4b3: serviceaccount "payload-processing-e2e-mt-7422b4b3" not found 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-7422b4b3-54dc98959 Error creating: pods "payload-pre-processing-e2e-mt-7422b4b3-54dc98959-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-7422b4b3: serviceaccount "payload-processing-e2e-mt-7422b4b3" not found 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-7422b4b3 Scaled up replica set payload-pre-processing-e2e-mt-7422b4b3-54dc98959 from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-7422b4b3-54dc98959 Created pod: payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-7422b4b3-7457ddb899 Created pod: payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc 39m Normal Started pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Container started 39m Normal Pulled pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Add eth0 [10.133.0.93/23] from ovn-kubernetes 39m Normal Created pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Container created 39m Normal Started pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Container started 39m Normal Created pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Container created 39m Normal Pulled pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Add eth0 [10.133.0.92/23] from ovn-kubernetes 39m Normal Killing pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf Stopping container istio-proxy 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-8ca52a-openshift-default New size: 2; reason: All metrics below target 39m Normal SuccessfulDelete replicaset/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 Deleted pod: e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf 39m Normal ScalingReplicaSet deployment/e2e-worker-w3-8ca52a-openshift-default Scaled down replica set e2e-worker-w3-8ca52a-openshift-default-6fb94bf447 from 3 to 2 39m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-516ccb-openshift-default Scaled up replica set e2e-shared-a-w0-516ccb-openshift-default-7745db7ff from 0 to 1 39m Normal Pulled pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff Created pod: e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv 39m Normal AddedLabel gateway/e2e-shared-a-w0-516ccb Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-516ccb 39m Normal AddedInterface pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Add eth0 [10.133.0.94/23] from ovn-kubernetes 39m Normal Started pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Container started 39m Normal Created pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Container created 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-781ccf27-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-781ccf27-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning Unhealthy pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Startup probe failed: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused 39m Warning ProbeError pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Startup probe error: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused... 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-516ccb Scaled up replica set payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9 from 0 to 1 39m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9 Error creating: pods "payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb: serviceaccount "payload-processing-e2e-shared-a-w0-516ccb" not found 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f Created pod: payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-516ccb Scaled up replica set payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9 Created pod: payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f Error creating: pods "payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-516ccb: serviceaccount "payload-processing-e2e-shared-a-w0-516ccb" not found 38m Normal Created pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Container created 38m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Container created 38m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Container started 38m Normal Pulled pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Add eth0 [10.133.0.96/23] from ovn-kubernetes 38m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Started pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Container started 38m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Add eth0 [10.133.0.95/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt 38m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 38m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt 38m Normal ScalingReplicaSet deployment/e2e-ait-3ef8140e-openshift-default Scaled up replica set e2e-ait-3ef8140e-openshift-default-6df8574985 from 1 to 4 38m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Container created 38m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Add eth0 [10.134.0.79/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm 38m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Container started 38m Normal ScalingReplicaSet deployment/e2e-mt-422c1029-openshift-default Scaled up replica set e2e-mt-422c1029-openshift-default-5786fb5789 from 0 to 1 38m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Add eth0 [10.133.0.98/23] from ovn-kubernetes 38m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Container created 38m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Container started 38m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Container started 38m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Container created 38m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Add eth0 [10.134.0.80/23] from ovn-kubernetes 38m Normal AddedLabel gateway/e2e-mt-422c1029 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-422c1029 38m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Add eth0 [10.133.0.99/23] from ovn-kubernetes 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-781ccf27-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Readiness probe failed: Get "http://10.132.0.173:15021/healthz/ready": read tcp 10.132.0.2:43984->10.132.0.173:15021: read: connection reset by peer 38m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Readiness probe error: Get "http://10.132.0.173:15021/healthz/ready": read tcp 10.132.0.2:43984->10.132.0.173:15021: read: connection reset by peer... 38m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Container started 38m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Container created 38m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Killing pod/payload-processing-e2e-ait-3ef8140e-bc85668dc-x67x5 Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-ait-3ef8140e-87bbbff55-k94cv Stopping container payload-pre-processing 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-781ccf27-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Startup probe failed: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused 38m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Startup probe error: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused... 38m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Startup probe error: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Startup probe failed: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused 38m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Startup probe failed: Get "http://10.132.0.173:15021/healthz/ready": dial tcp 10.132.0.173:15021: connect: connection refused 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-422c1029 Scaled up replica set payload-processing-e2e-mt-422c1029-769464bdcc from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-422c1029 Scaled up replica set payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 from 0 to 1 38m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Startup probe error: Get "http://10.132.0.173:15021/healthz/ready": dial tcp 10.132.0.173:15021: connect: connection refused... 38m Warning FailedCreate replicaset/payload-processing-e2e-mt-422c1029-769464bdcc Error creating: pods "payload-processing-e2e-mt-422c1029-769464bdcc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-422c1029: serviceaccount "payload-processing-e2e-mt-422c1029" not found 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 Error creating: pods "payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-422c1029: serviceaccount "payload-processing-e2e-mt-422c1029" not found 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 Created pod: payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-422c1029-769464bdcc Created pod: payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 38m Normal AddedInterface pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Add eth0 [10.134.0.81/23] from ovn-kubernetes 38m Normal Pulled pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Add eth0 [10.133.0.100/23] from ovn-kubernetes 38m Normal Created pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Container created 38m Normal Pulled pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Container created 38m Normal Started pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Container started 38m Normal Started pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Container started 38m Normal Killing pod/payload-processing-e2e-derive-781ccf27-8698b866fc-4p7d6 Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-derive-781ccf27-675589c8f9-kb4ms Stopping container payload-pre-processing 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled down replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 10 to 7 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 7; reason: All metrics below target 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d 38m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-b4j9d Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 Stopping container istio-proxy 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-l9z92 38m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-zcf5p Stopping container istio-proxy 38m Normal Pulled pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Container created 38m Normal Started pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Container started 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-7422b4b3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-7422b4b3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning Unhealthy pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Startup probe failed: Get "http://10.133.0.90:15021/healthz/ready": dial tcp 10.133.0.90:15021: connect: connection refused 38m Warning ProbeError pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Startup probe error: Get "http://10.133.0.90:15021/healthz/ready": dial tcp 10.133.0.90:15021: connect: connection refused... 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Stopping container istio-proxy 38m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled down replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 6 to 5 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 5; reason: All metrics below target 38m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst 38m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l 38m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst Stopping container istio-proxy 38m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-6zk2l Stopping container istio-proxy 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 5; reason: All metrics below target 38m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled down replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 7 to 5 38m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Readiness probe failed: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused 38m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 Readiness probe error: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused... 37m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx 37m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g 37m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 37m Normal SuccessfulCreate replicaset/e2e-ait-3ef8140e-openshift-default-6df8574985 Created pod: e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff 37m Normal ScalingReplicaSet deployment/e2e-ait-3ef8140e-openshift-default Scaled up replica set e2e-ait-3ef8140e-openshift-default-6df8574985 from 4 to 8 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ait-3ef8140e-openshift-default New size: 8; reason: All metrics below target 37m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Add eth0 [10.133.0.101/23] from ovn-kubernetes 37m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Container created 37m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Add eth0 [10.134.0.83/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Container started 37m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Add eth0 [10.132.0.186/23] from ovn-kubernetes 37m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Container started 37m Normal AddedInterface pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Add eth0 [10.132.0.185/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Container created 37m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Container started 37m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Container created 37m Normal Pulled pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Container created 37m Normal Started pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Container started 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 5; reason: All metrics below target 37m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Stopping container istio-proxy 37m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled down replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 10 to 5 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl 37m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 Stopping container istio-proxy 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx 37m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-5sszl Stopping container istio-proxy 37m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 37m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk 37m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs 37m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm 37m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 6; reason: All metrics below target 37m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled down replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 10 to 6 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 6; reason: All metrics below target 37m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Stopping container istio-proxy 37m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled down replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 10 to 6 37m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Stopping container istio-proxy 37m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 37m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs 37m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj 37m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 37m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t6fpm Stopping container istio-proxy 37m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Stopping container istio-proxy 37m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Startup probe error: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-9dvkj Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Startup probe failed: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Stopping container istio-proxy 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Startup probe failed: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Stopping container istio-proxy 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Startup probe failed: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Stopping container istio-proxy 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sqc7g Startup probe error: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused... 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Startup probe failed: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-vkmgx Startup probe error: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused... 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Stopping container istio-proxy 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-xdz98 Startup probe error: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused... 37m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal Killing pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Stopping container istio-proxy 37m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mx8z6 Readiness probe failed: HTTP probe failed with statuscode: 503 37m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf Created pod: e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-7422b4b3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-hlzrs Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-g4n29 Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-41e4de-openshift-default Scaled up replica set e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf from 0 to 1 37m Normal AddedLabel gateway/e2e-shared-a-w6-41e4de Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-41e4de 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-n9ztt Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-7422b4b3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Readiness probe error: HTTP probe failed with statuscode: 503... 37m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Readiness probe error: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk Readiness probe failed: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-llqf9 Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal AddedInterface pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Add eth0 [10.132.0.187/23] from ovn-kubernetes 37m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Readiness probe failed: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Readiness probe error: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx Readiness probe failed: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-wxwbb Readiness probe error: Get "http://10.132.0.126:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-ait-3ef8140e-openshift-default-6df8574985-sv8pk Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal Killing pod/e2e-derive-781ccf27-openshift-default-79c47976f-24n2d Stopping container istio-proxy 37m Normal AddedLabel gateway/e2e-shared-a-w2-b55a43 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-b55a43 37m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-b55a43-openshift-default Scaled up replica set e2e-shared-a-w2-b55a43-openshift-default-b68dbf55 from 0 to 1 37m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55 Created pod: e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b 37m Normal AddedInterface pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Add eth0 [10.132.0.188/23] from ovn-kubernetes 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-41e4de Scaled up replica set payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9 from 0 to 1 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-41e4de Scaled up replica set payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc from 0 to 1 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9 Created pod: payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc Created pod: payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc Error creating: pods "payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de: serviceaccount "payload-processing-e2e-shared-a-w6-41e4de" not found 37m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9 Error creating: pods "payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-41e4de: serviceaccount "payload-processing-e2e-shared-a-w6-41e4de" not found 37m Normal Created pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Container created 37m Normal Started pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Container started 37m Normal Pulled pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Add eth0 [10.132.0.190/23] from ovn-kubernetes 37m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Container started 37m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Container created 37m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Add eth0 [10.132.0.189/23] from ovn-kubernetes 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-b55a43 Scaled up replica set payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5 from 0 to 1 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c Error creating: pods "payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43: serviceaccount "payload-processing-e2e-shared-a-w2-b55a43" not found 37m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5 Error creating: pods "payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-b55a43: serviceaccount "payload-processing-e2e-shared-a-w2-b55a43" not found 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-b55a43 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c from 0 to 1 37m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5 Created pod: payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k 37m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Container started 37m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Container created 37m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Add eth0 [10.132.0.192/23] from ovn-kubernetes 37m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Add eth0 [10.132.0.191/23] from ovn-kubernetes 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c Created pod: payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g 37m Normal AddedLabel gateway/e2e-api-a-512caa Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-512caa 37m Normal Created pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Container created 37m Normal Pulled pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal Started pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Container started 37m Normal AddedInterface pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Add eth0 [10.132.0.193/23] from ovn-kubernetes 37m Normal ScalingReplicaSet deployment/e2e-api-a-512caa-openshift-default Scaled up replica set e2e-api-a-512caa-openshift-default-65d4749b7f from 0 to 1 37m Normal SuccessfulCreate replicaset/e2e-api-a-512caa-openshift-default-65d4749b7f Created pod: e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf 37m Normal Pulled pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Container started 37m Normal Created pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Container created 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-512caa Scaled up replica set payload-processing-e2e-api-a-512caa-6946957b9 from 0 to 1 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-512caa Scaled up replica set payload-pre-processing-e2e-api-a-512caa-f685cdfc9 from 0 to 1 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-512caa-f685cdfc9 Error creating: pods "payload-pre-processing-e2e-api-a-512caa-f685cdfc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-512caa: serviceaccount "payload-processing-e2e-api-a-512caa" not found 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-512caa-f685cdfc9 Created pod: payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv 37m Warning FailedCreate replicaset/payload-processing-e2e-api-a-512caa-6946957b9 Error creating: pods "payload-processing-e2e-api-a-512caa-6946957b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-512caa: serviceaccount "payload-processing-e2e-api-a-512caa" not found 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-512caa-6946957b9 Created pod: payload-processing-e2e-api-a-512caa-6946957b9-9zspt 37m Normal Started pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Container started 37m Normal Pulled pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal Created pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Container created 37m Normal Started pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Container started 37m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Add eth0 [10.132.0.194/23] from ovn-kubernetes 37m Normal Created pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Container created 37m Normal Pulled pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Add eth0 [10.132.0.195/23] from ovn-kubernetes 37m Warning Unhealthy pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Startup probe failed: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused 37m Warning ProbeError pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Startup probe error: Get "http://10.133.0.94:15021/healthz/ready": dial tcp 10.133.0.94:15021: connect: connection refused... 37m Normal AddedLabel gateway/e2e-shared-a-w3-9b614c Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-9b614c 37m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-9b614c-openshift-default Scaled up replica set e2e-shared-a-w3-9b614c-openshift-default-7955dc4547 from 0 to 1 37m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547 Created pod: e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd 37m Normal AddedInterface pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Add eth0 [10.132.0.196/23] from ovn-kubernetes 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-516ccb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-516ccb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c: serviceaccount "payload-processing-e2e-shared-a-w3-9b614c" not found 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967 Created pod: payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-9b614c Scaled up replica set payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967 from 0 to 1 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-9b614c Scaled up replica set payload-processing-e2e-shared-a-w3-9b614c-c47575765 from 0 to 1 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-9b614c-c47575765 Created pod: payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn 36m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-9b614c-c47575765 Error creating: pods "payload-processing-e2e-shared-a-w3-9b614c-c47575765-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c: serviceaccount "payload-processing-e2e-shared-a-w3-9b614c" not found 36m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Add eth0 [10.134.0.85/23] from ovn-kubernetes 36m Normal Started pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Container started 36m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Add eth0 [10.134.0.86/23] from ovn-kubernetes 36m Normal Pulled pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Created pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Container created 36m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Container started 36m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Container created 36m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Readiness probe error: Get "http://10.133.0.99:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Readiness probe failed: Get "http://10.133.0.99:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Container created 36m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Container started 36m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-c3d917-openshift-default Scaled up replica set e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767 from 0 to 1 36m Normal AddedLabel gateway/e2e-shared-b-w0-c3d917 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-c3d917 36m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767 Created pod: e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg 36m Normal AddedInterface pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Add eth0 [10.132.0.197/23] from ovn-kubernetes 36m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Startup probe error: Get "http://10.133.0.99:15021/healthz/ready": dial tcp 10.133.0.99:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Startup probe failed: Get "http://10.133.0.99:15021/healthz/ready": dial tcp 10.133.0.99:15021: connect: connection refused 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-c3d917 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f from 0 to 1 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f Error creating: pods "payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917: serviceaccount "payload-processing-e2e-shared-b-w0-c3d917" not found 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-c3d917 Scaled up replica set payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c from 0 to 1 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c Created pod: payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 36m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c Error creating: pods "payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-c3d917: serviceaccount "payload-processing-e2e-shared-b-w0-c3d917" not found 36m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Add eth0 [10.132.0.198/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f Created pod: payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 36m Normal Pulled pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Created pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Container created 36m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Add eth0 [10.133.0.102/23] from ovn-kubernetes 36m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Container started 36m Normal Started pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Container started 36m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Container created 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-422c1029-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Stopping container istio-proxy 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-422c1029-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 6; reason: All metrics below target 36m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Stopping container istio-proxy 36m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled down replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 10 to 6 36m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Stopping container istio-proxy 36m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn 36m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x 36m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 36m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr 36m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-qh2mn Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-44bfr Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-2w82x Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-hbzx5 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal Pulled pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Container created 36m Normal Started pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Container started 35m Normal Killing pod/payload-processing-e2e-mt-7422b4b3-7457ddb899-2q5xc Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-mt-7422b4b3-54dc98959-95gzd Stopping container payload-pre-processing 35m Warning ProbeError pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Startup probe error: Get "http://10.132.0.187:15021/healthz/ready": dial tcp 10.132.0.187:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Startup probe failed: Get "http://10.132.0.187:15021/healthz/ready": dial tcp 10.132.0.187:15021: connect: connection refused 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-41e4de-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-41e4de-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-41e4de-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-41e4de-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled down replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 6 to 5 35m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs 35m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 35m Normal Started pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Container started 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 5; reason: All metrics below target 35m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Stopping container istio-proxy 35m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Stopping container istio-proxy 35m Normal Created pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Container created 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 4; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default Scaled down replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 5 to 4 35m Normal Pulled pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled down replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 6 to 5 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 5; reason: All metrics below target 35m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-kkjp5 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm 35m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-6ccgs Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-c2cxq Stopping container istio-proxy 35m Warning Unhealthy pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Startup probe failed: Get "http://10.132.0.193:15021/healthz/ready": dial tcp 10.132.0.193:15021: connect: connection refused 35m Warning ProbeError pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Startup probe error: Get "http://10.132.0.193:15021/healthz/ready": dial tcp 10.132.0.193:15021: connect: connection refused... 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-512caa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-512caa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal Killing pod/e2e-mt-7422b4b3-openshift-default-69d88b745c-48v2w Stopping container istio-proxy 35m Normal SuccessfulCreate replicaset/e2e-api-a-81f37a-openshift-default-68b6d69789 Created pod: e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr 35m Normal ScalingReplicaSet deployment/e2e-api-a-81f37a-openshift-default Scaled up replica set e2e-api-a-81f37a-openshift-default-68b6d69789 from 0 to 1 35m Normal AddedLabel gateway/e2e-api-a-81f37a Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-81f37a 35m Normal AddedInterface pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Add eth0 [10.133.0.107/23] from ovn-kubernetes 35m Normal AddedLabel gateway/e2e-shared-b-w6-24aa4c Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-24aa4c 35m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 Stopping container discovery 35m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-pgvh7 35m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 35m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7 Created pod: e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z 35m Normal AddedInterface pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Add eth0 [10.134.0.89/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-24aa4c-openshift-default Scaled up replica set e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7 from 0 to 1 35m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 35m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default New size: 3; reason: All metrics below target 35m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 4; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default Scaled down replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 5 to 4 35m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-81f37a-9454c4d75 Created pod: payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr 35m Warning FailedCreate replicaset/payload-processing-e2e-api-a-81f37a-6b45df848c Error creating: pods "payload-processing-e2e-api-a-81f37a-6b45df848c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-81f37a: serviceaccount "payload-processing-e2e-api-a-81f37a" not found 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-81f37a-9454c4d75 Error creating: pods "payload-pre-processing-e2e-api-a-81f37a-9454c4d75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-81f37a: serviceaccount "payload-processing-e2e-api-a-81f37a" not found 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-81f37a-6b45df848c Created pod: payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-81f37a Scaled up replica set payload-pre-processing-e2e-api-a-81f37a-9454c4d75 from 0 to 1 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-81f37a Scaled up replica set payload-processing-e2e-api-a-81f37a-6b45df848c from 0 to 1 35m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Add eth0 [10.134.0.90/23] from ovn-kubernetes 35m Normal Pulled pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Created pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Container created 35m Normal Started pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Container started 35m Normal Pulled pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-9hjxv Stopping container istio-proxy 35m Normal AddedInterface pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Add eth0 [10.133.0.108/23] from ovn-kubernetes 35m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-vlb67 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal Started pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Container started 35m Normal Created pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Container created 34m Normal AddedLabel gateway/e2e-shared-b-w2-4511b0 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-4511b0 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc Error creating: pods "payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c: serviceaccount "payload-processing-e2e-shared-b-w6-24aa4c" not found 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc Created pod: payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-24aa4c Scaled up replica set payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc from 0 to 1 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5 Error creating: pods "payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c: serviceaccount "payload-processing-e2e-shared-b-w6-24aa4c" not found 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5 Created pod: payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-24aa4c Scaled up replica set payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5 from 0 to 1 34m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Container created 34m Normal Pulled pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal AddedInterface pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Add eth0 [10.134.0.93/23] from ovn-kubernetes 34m Normal Started pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Container started 34m Normal Created pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Container created 34m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Add eth0 [10.134.0.91/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc Created pod: e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr 34m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-4511b0-openshift-default Scaled up replica set e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc from 0 to 1 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Add eth0 [10.134.0.92/23] from ovn-kubernetes 34m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Container started 34m Warning ProbeError pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Readiness probe error: Get "http://10.132.0.188:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Warning Unhealthy pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Readiness probe failed: Get "http://10.132.0.188:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Normal Pulled pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Container started 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495 Error creating: pods "payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0: serviceaccount "payload-processing-e2e-shared-b-w2-4511b0" not found 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf Error creating: pods "payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-4511b0: serviceaccount "payload-processing-e2e-shared-b-w2-4511b0" not found 34m Normal Pulled pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Container created 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-4511b0 Scaled up replica set payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-4511b0 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495 from 0 to 1 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495 Created pod: payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh 34m Normal Started pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Container started 34m Normal Pulled pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Container created 34m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Add eth0 [10.133.0.109/23] from ovn-kubernetes 34m Normal Created pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Container created 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Add eth0 [10.133.0.110/23] from ovn-kubernetes 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf Created pod: payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z 34m Normal Started pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Container started 34m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Container created 34m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Container started 34m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-9b614c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-9b614c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-512caa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-512caa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning ProbeError pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Startup probe error: Get "http://10.132.0.196:15021/healthz/ready": dial tcp 10.132.0.196:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Startup probe failed: Get "http://10.132.0.196:15021/healthz/ready": dial tcp 10.132.0.196:15021: connect: connection refused 34m Warning Unhealthy pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Startup probe failed: Get "http://10.132.0.188:15021/healthz/ready": dial tcp 10.132.0.188:15021: connect: connection refused 34m Warning ProbeError pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Startup probe error: Get "http://10.132.0.188:15021/healthz/ready": dial tcp 10.132.0.188:15021: connect: connection refused... 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-b55a43-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-b55a43-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal AddedLabel gateway/e2e-api-b-ff338b Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-ff338b 34m Normal ScalingReplicaSet deployment/e2e-api-b-ff338b-openshift-default Scaled up replica set e2e-api-b-ff338b-openshift-default-84498fd4f9 from 0 to 1 34m Normal SuccessfulCreate replicaset/e2e-api-b-ff338b-openshift-default-84498fd4f9 Created pod: e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 34m Normal AddedInterface pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Add eth0 [10.133.0.111/23] from ovn-kubernetes 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-ff338b Scaled up replica set payload-processing-e2e-api-b-ff338b-6799d5d75c from 0 to 1 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-ff338b Scaled up replica set payload-pre-processing-e2e-api-b-ff338b-7454f49cff from 0 to 1 34m Warning FailedCreate replicaset/payload-processing-e2e-api-b-ff338b-6799d5d75c Error creating: pods "payload-processing-e2e-api-b-ff338b-6799d5d75c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-ff338b: serviceaccount "payload-processing-e2e-api-b-ff338b" not found 34m Normal Pulled pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-ff338b-6799d5d75c Created pod: payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-ff338b-7454f49cff Error creating: pods "payload-pre-processing-e2e-api-b-ff338b-7454f49cff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-ff338b: serviceaccount "payload-processing-e2e-api-b-ff338b" not found 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-ff338b-7454f49cff Created pod: payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt 34m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Add eth0 [10.133.0.112/23] from ovn-kubernetes 34m Normal Created pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Container created 34m Normal Pulled pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal AddedInterface pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Add eth0 [10.134.0.94/23] from ovn-kubernetes 34m Normal Started pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Container started 34m Normal Started pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Container started 34m Normal Created pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Container created 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-b55a43-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-b55a43-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-9b614c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-9b614c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f Created pod: e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm 34m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-7cfd50-openshift-default Scaled up replica set e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f from 0 to 1 34m Normal AddedLabel gateway/e2e-shared-b-w3-7cfd50 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-7cfd50 34m Normal AddedInterface pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Add eth0 [10.132.0.200/23] from ovn-kubernetes 34m Normal Created pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Container created 34m Normal Pulled pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Container started 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-c3d917-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-c3d917-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning ProbeError pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Startup probe error: Get "http://10.132.0.197:15021/healthz/ready": dial tcp 10.132.0.197:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Startup probe failed: Get "http://10.132.0.197:15021/healthz/ready": dial tcp 10.132.0.197:15021: connect: connection refused 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479 Error creating: pods "payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50: serviceaccount "payload-processing-e2e-shared-b-w3-7cfd50" not found 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-7cfd50 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479 from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-7cfd50 Scaled up replica set payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79 from 0 to 1 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79 Error creating: pods "payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-7cfd50: serviceaccount "payload-processing-e2e-shared-b-w3-7cfd50" not found 33m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Container created 33m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Container started 33m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Add eth0 [10.133.0.113/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479 Created pod: payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79 Created pod: payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Add eth0 [10.134.0.95/23] from ovn-kubernetes 33m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Started pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Container started 33m Normal Pulled pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Container created 33m Normal ScalingReplicaSet deployment/e2e-inf-a-18edc4-openshift-default Scaled up replica set e2e-inf-a-18edc4-openshift-default-7f884f6765 from 0 to 1 33m Normal SuccessfulCreate replicaset/e2e-inf-a-18edc4-openshift-default-7f884f6765 Created pod: e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 33m Normal AddedLabel gateway/e2e-inf-a-18edc4 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-18edc4 33m Normal AddedInterface pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Add eth0 [10.134.0.96/23] from ovn-kubernetes 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-18edc4 Scaled up replica set payload-processing-e2e-inf-a-18edc4-69958dfcd7 from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-18edc4 Scaled up replica set payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7 from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-18edc4-69958dfcd7 Created pod: payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc 33m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-18edc4-69958dfcd7 Error creating: pods "payload-processing-e2e-inf-a-18edc4-69958dfcd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-18edc4: serviceaccount "payload-processing-e2e-inf-a-18edc4" not found 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7 Created pod: payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7 Error creating: pods "payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-18edc4: serviceaccount "payload-processing-e2e-inf-a-18edc4" not found 33m Normal Created pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Container created 33m Normal Created pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Container created 33m Normal Pulled pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Pulled pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Started pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Container started 33m Normal AddedInterface pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Add eth0 [10.133.0.115/23] from ovn-kubernetes 33m Normal Started pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Container started 33m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Add eth0 [10.133.0.114/23] from ovn-kubernetes 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-c3d917-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-c3d917-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 4; reason: All metrics below target 33m Normal Killing pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Stopping container payload-processing 33m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs Stopping container istio-proxy 33m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled down replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 5 to 4 33m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-c9bzs 33m Normal Killing pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Stopping container payload-pre-processing 33m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Stopping container istio-proxy 33m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 4; reason: All metrics below target 33m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 4; reason: All metrics below target 33m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Stopping container istio-proxy 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default New size: 3; reason: All metrics below target 33m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled down replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 5 to 4 33m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled down replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 5 to 4 33m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc 33m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Stopping container istio-proxy 33m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d9l8v Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-mt9xw Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal Started pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Container started 32m Normal Created pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Container created 32m Normal Pulled pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Warning Unhealthy pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Readiness probe failed: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused 32m Warning ProbeError pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Readiness probe error: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused... 32m Normal Started pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Container started 32m Normal Created pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Container created 32m Normal Pulled pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Warning ProbeError pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Startup probe error: Get "http://10.134.0.89:15021/healthz/ready": dial tcp 10.134.0.89:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Startup probe failed: Get "http://10.134.0.89:15021/healthz/ready": dial tcp 10.134.0.89:15021: connect: connection refused 32m Warning Unhealthy pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Startup probe failed: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused 32m Warning ProbeError pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Startup probe error: Get "http://10.133.0.107:15021/healthz/ready": dial tcp 10.133.0.107:15021: connect: connection refused... 32m Warning Unhealthy pod/payload-processing-e2e-mt-422c1029-769464bdcc-rhtt8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-81f37a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-81f37a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-24aa4c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-24aa4c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-81f37a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-81f37a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-24aa4c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-24aa4c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 32m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 5; reason: All metrics below target 32m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Stopping container istio-proxy 32m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-v7ph2 Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled down replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 6 to 5 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 3; reason: All metrics below target 32m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default Scaled down replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 4 to 3 32m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 32m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Stopping container istio-proxy 32m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-sflfk Stopping container istio-proxy 32m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-mzffc Stopping container istio-proxy 32m Normal Pulled pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Container created 32m Normal Started pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Container started 32m Normal Pulled pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Container created 32m Normal Started pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Container started 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-4511b0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-4511b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 4; reason: All metrics below target 32m Warning Unhealthy pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Startup probe failed: Get "http://10.134.0.93:15021/healthz/ready": dial tcp 10.134.0.93:15021: connect: connection refused 32m Warning ProbeError pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Startup probe error: Get "http://10.133.0.111:15021/healthz/ready": dial tcp 10.133.0.111:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Startup probe failed: Get "http://10.133.0.111:15021/healthz/ready": dial tcp 10.133.0.111:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Startup probe error: Get "http://10.134.0.93:15021/healthz/ready": dial tcp 10.134.0.93:15021: connect: connection refused... 32m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Stopping container istio-proxy 32m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 32m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled down replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 5 to 4 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-ff338b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 3; reason: All metrics below target 32m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-ff338b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default Scaled down replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 4 to 3 32m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Stopping container istio-proxy 32m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Normal Killing pod/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-api-a-81f37a-6b45df848c-rtwbr Stopping container payload-processing 31m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-41e4de-7cb6494bc-j9v96 Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-shared-a-w6-41e4de-7b5864bdf9-zgvdb Stopping container payload-processing 31m Normal Killing pod/payload-processing-e2e-shared-a-w2-b55a43-7b869cc5d5-dcf4k Stopping container payload-processing 31m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-b55a43-884b5fd6c-qsk7g Stopping container payload-pre-processing 31m Normal Pulled pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Container created 31m Normal Started pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Container started 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-4511b0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-4511b0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning ProbeError pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Startup probe error: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused... 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-7cfd50-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning Unhealthy pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Startup probe failed: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-7cfd50-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-ff338b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-ff338b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal Killing pod/payload-pre-processing-e2e-api-a-512caa-f685cdfc9-4wphv Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-api-a-512caa-6946957b9-9zspt Stopping container payload-processing 31m Warning ProbeError pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Readiness probe error: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Readiness probe failed: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused 31m Normal Created pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Container created 31m Normal Started pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Container started 31m Normal Pulled pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Warning ProbeError pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Startup probe error: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Startup probe failed: Get "http://10.134.0.96:15021/healthz/ready": dial tcp 10.134.0.96:15021: connect: connection refused 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-7cfd50-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-7cfd50-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-18edc4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-18edc4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Normal Killing pod/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml Stopping container payload-pre-processing 30m Normal Killing pod/e2e-shared-a-w3-9b614c-openshift-default-7955dc4547-4gpxd Stopping container istio-proxy 30m Normal Killing pod/payload-processing-e2e-inf-a-18edc4-69958dfcd7-db6vc Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-inf-a-18edc4-5944d9c6d7-5rq52 Stopping container payload-pre-processing 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-18edc4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-18edc4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-shared-b-w3-7cfd50-bddc88f79-lgtk5 Stopping container payload-processing 30m Normal Killing pod/e2e-shared-a-w6-41e4de-openshift-default-9fc9dd7bf-29vmq Stopping container istio-proxy 30m Normal Killing pod/e2e-api-a-81f37a-openshift-default-68b6d69789-k4kvr Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 Stopping container istio-proxy 30m Normal Killing pod/e2e-shared-a-w2-b55a43-openshift-default-b68dbf55-jn82b Stopping container istio-proxy 30m Normal Killing pod/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn Stopping container payload-pre-processing 29m Warning FailedCreate replicaset/payload-processing-e2e-mt-422c1029-769464bdcc Error creating: pods "payload-processing-e2e-mt-422c1029-769464bdcc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-422c1029: serviceaccount "payload-processing-e2e-mt-422c1029" not found 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-422c1029-769464bdcc Created pod: payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 Error creating: pods "payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-422c1029: serviceaccount "payload-processing-e2e-mt-422c1029" not found 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 Created pod: payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-422c1029 Scaled up replica set payload-processing-e2e-mt-422c1029-769464bdcc from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-422c1029 Scaled up replica set payload-pre-processing-e2e-mt-422c1029-5cc6f68c9 from 0 to 1 29m Normal Started pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Container started 29m Normal Pulled pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Container created 29m Normal Created pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Container created 29m Normal AddedInterface pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Add eth0 [10.134.0.101/23] from ovn-kubernetes 29m Normal Started pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Container started 29m Normal AddedInterface pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Add eth0 [10.134.0.102/23] from ovn-kubernetes 29m Normal Pulled pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Killing pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Stopping container payload-pre-processing 29m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-4511b0-5b9bbc7495-xq2bh Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-shared-b-w2-4511b0-7b9799bbbf-65s8z Stopping container payload-processing 29m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Stopping container istio-proxy 29m Normal Killing pod/payload-processing-e2e-shared-a-w0-516ccb-6597c8c5d9-2zp7b Stopping container payload-processing 29m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled down replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 4 to 3 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 3; reason: All metrics below target 29m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-516ccb-6654c67c7f-7dkpk Stopping container payload-pre-processing 29m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 3; reason: All metrics below target 29m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default Scaled down replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 4 to 3 29m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg 29m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w2-d8837a-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-d8837a-openshift-default-66b7f49787 from 3 to 2 29m Normal SuccessfulDelete replicaset/e2e-worker-w2-d8837a-openshift-default-66b7f49787 Deleted pod: e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-d8837a-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-9cff60-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 29m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w4-bce513-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w4-bce513-openshift-default-6645cfc867 from 3 to 2 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bce513-openshift-default New size: 2; reason: All metrics below target 29m Warning ProbeError pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Readiness probe error: HTTP probe failed with statuscode: 503... 29m Normal SuccessfulDelete replicaset/e2e-worker-w1-9cff60-openshift-default-d6667c5f9 Deleted pod: e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr 29m Normal ScalingReplicaSet deployment/e2e-worker-w1-9cff60-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-9cff60-openshift-default-d6667c5f9 from 3 to 2 29m Normal SuccessfulDelete replicaset/e2e-worker-w4-bce513-openshift-default-6645cfc867 Deleted pod: e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg 29m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Stopping container istio-proxy 29m Warning Unhealthy pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-h2bzr Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qs5cg Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-n2gxh Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/payload-processing-e2e-worker-w4-bce513-556ddf7445-hz787 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-4m2dg Readiness probe error: HTTP probe failed with statuscode: 503... 29m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-67bf986f99 (combined from similar events): Deleted pod: maas-default-gateway-openshift-default-67bf986f99-8kz9s 29m Normal Killing pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Stopping container istio-proxy 29m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 29m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-bce513-5fd46c7dff-t4bjw Readiness probe failed: timeout: failed to connect service "10.133.0.40:50052" within 1s: context deadline exceeded 29m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-8kz9s Readiness probe error: HTTP probe failed with statuscode: 503... 29m Normal Killing pod/e2e-shared-a-w0-516ccb-openshift-default-7745db7ff-4qwmv Stopping container istio-proxy 29m Normal Killing pod/e2e-shared-b-w3-7cfd50-openshift-default-5b986d64f-ggrzm Stopping container istio-proxy 29m Normal SuccessfulCreate replicaset/e2e-api-a-d67bf9-openshift-default-7585d64b69 Created pod: e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc 29m Normal ScalingReplicaSet deployment/e2e-api-a-d67bf9-openshift-default Scaled up replica set e2e-api-a-d67bf9-openshift-default-7585d64b69 from 0 to 1 29m Normal AddedLabel gateway/e2e-api-a-d67bf9 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-d67bf9 29m Normal AddedInterface pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Add eth0 [10.132.0.217/23] from ovn-kubernetes 29m Warning FailedCreate replicaset/payload-processing-e2e-api-a-d67bf9-6fb7f9c547 Error creating: pods "payload-processing-e2e-api-a-d67bf9-6fb7f9c547-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d67bf9: serviceaccount "payload-processing-e2e-api-a-d67bf9" not found 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-d67bf9 Scaled up replica set payload-processing-e2e-api-a-d67bf9-6fb7f9c547 from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-d67bf9 Scaled up replica set payload-pre-processing-e2e-api-a-d67bf9-6b574779b7 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7 Created pod: payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7 Error creating: pods "payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-d67bf9: serviceaccount "payload-processing-e2e-api-a-d67bf9" not found 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-d67bf9-6fb7f9c547 Created pod: payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz 29m Normal Started pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Container started 29m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Add eth0 [10.132.0.218/23] from ovn-kubernetes 29m Normal Pulled pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Container created 29m Normal AddedInterface pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Add eth0 [10.132.0.219/23] from ovn-kubernetes 29m Normal Started pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Container started 29m Normal Created pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Container created 29m Normal Pulled pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Killing pod/payload-processing-e2e-shared-b-w0-c3d917-7b5844dd7c-5wjk6 Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-c3d917-7b6674bc7f-7qph4 Stopping container payload-pre-processing 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d67bf9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d67bf9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Normal Killing pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z Stopping container istio-proxy 28m Normal Killing pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr Stopping container istio-proxy 28m Normal Killing pod/e2e-api-a-512caa-openshift-default-65d4749b7f-f4hlf Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Stopping container istio-proxy 28m Normal Killing pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-dgsrb Stopping container istio-proxy 28m Warning ProbeError pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning Unhealthy pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj Readiness probe failed: HTTP probe failed with statuscode: 503 28m Normal Killing pod/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-api-b-ff338b-7454f49cff-rjbbt Stopping container payload-pre-processing 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-6a0a09-openshift-default New size: 2; reason: All metrics below target 28m Normal ScalingReplicaSet deployment/e2e-worker-w0-6a0a09-openshift-default Scaled down replica set e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 from 3 to 2 28m Normal SuccessfulDelete replicaset/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7 Deleted pod: e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px 28m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-vr5px Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/e2e-worker-w6-2182a5-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-2182a5-openshift-default-89fd49f79 from 3 to 2 28m Normal SuccessfulDelete replicaset/e2e-worker-w6-2182a5-openshift-default-89fd49f79 Deleted pod: e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-2182a5-openshift-default New size: 2; reason: All metrics below target 28m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Stopping container istio-proxy 28m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-d2hmh Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal Killing pod/e2e-shared-b-w0-c3d917-openshift-default-67f9f9c767-spgpg Stopping container istio-proxy 28m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Container created 28m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Container started 28m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-7h99v 28m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Container created 28m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Container started 28m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Add eth0 [10.133.0.119/23] from ovn-kubernetes 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-422c1029-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 28m Normal ScalingReplicaSet deployment/e2e-mt-422c1029-openshift-default Scaled up replica set e2e-mt-422c1029-openshift-default-5786fb5789 from 1 to 4 28m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-99f4v 28m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Add eth0 [10.134.0.105/23] from ovn-kubernetes 28m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs 28m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Add eth0 [10.132.0.223/23] from ovn-kubernetes 28m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Container created 28m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Container started 28m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 27m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default (combined from similar events): Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 4 to 8 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Container created 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Container started 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Container created 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Container started 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Container created 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Add eth0 [10.133.0.120/23] from ovn-kubernetes 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Add eth0 [10.132.0.224/23] from ovn-kubernetes 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Add eth0 [10.133.0.121/23] from ovn-kubernetes 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Container started 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Add eth0 [10.134.0.106/23] from ovn-kubernetes 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Container started 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Container created 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Startup probe failed: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Startup probe error: Get "http://10.133.0.119:15021/healthz/ready": dial tcp 10.133.0.119:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Startup probe failed: Get "http://10.132.0.223:15021/healthz/ready": dial tcp 10.132.0.223:15021: connect: connection refused 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Startup probe error: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Startup probe error: Get "http://10.132.0.223:15021/healthz/ready": dial tcp 10.132.0.223:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Startup probe failed: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused 27m Normal Killing pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Stopping container payload-pre-processing 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Startup probe error: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused... 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Startup probe error: Get "http://10.133.0.120:15021/healthz/ready": dial tcp 10.133.0.120:15021: connect: connection refused... 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Startup probe error: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused... 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Startup probe failed: Get "http://10.133.0.120:15021/healthz/ready": dial tcp 10.133.0.120:15021: connect: connection refused 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Startup probe failed: Get "http://10.132.0.224:15021/healthz/ready": dial tcp 10.132.0.224:15021: connect: connection refused 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Startup probe error: Get "http://10.132.0.224:15021/healthz/ready": dial tcp 10.132.0.224:15021: connect: connection refused... 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Startup probe failed: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Startup probe failed: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-74nsh 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-n26pm 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-zct7m 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-422c1029-openshift-default New size: 8; reason: 27m Normal ScalingReplicaSet deployment/e2e-mt-422c1029-openshift-default Scaled up replica set e2e-mt-422c1029-openshift-default-5786fb5789 from 4 to 8 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Add eth0 [10.133.0.122/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Add eth0 [10.132.0.226/23] from ovn-kubernetes 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Container started 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Container created 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Container started 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Container created 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Container created 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Add eth0 [10.134.0.108/23] from ovn-kubernetes 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Container created 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Container started 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Add eth0 [10.132.0.227/23] from ovn-kubernetes 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Container started 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default (combined from similar events): New size: 10; reason: 27m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-67bf986f99 from 8 to 10 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Add eth0 [10.134.0.109/23] from ovn-kubernetes 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Container started 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Container created 27m Normal Started pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Container started 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Startup probe failed: Get "http://10.134.0.108:15021/healthz/ready": dial tcp 10.134.0.108:15021: connect: connection refused 27m Normal AddedInterface pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Add eth0 [10.133.0.123/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Startup probe error: Get "http://10.134.0.108:15021/healthz/ready": dial tcp 10.134.0.108:15021: connect: connection refused... 27m Normal Pulled pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Container created 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Startup probe failed: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Startup probe error: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused... 27m Normal Killing pod/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj Stopping container payload-pre-processing 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Startup probe error: Get "http://10.132.0.227:15021/healthz/ready": dial tcp 10.132.0.227:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Startup probe failed: Get "http://10.132.0.227:15021/healthz/ready": dial tcp 10.132.0.227:15021: connect: connection refused 27m Normal Killing pod/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd Stopping container payload-processing 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Startup probe error: Get "http://10.132.0.226:15021/healthz/ready": dial tcp 10.132.0.226:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Startup probe failed: Get "http://10.132.0.226:15021/healthz/ready": dial tcp 10.132.0.226:15021: connect: connection refused 27m Normal Killing pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Stopping container istio-proxy 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Startup probe failed: Get "http://10.133.0.123:15021/healthz/ready": dial tcp 10.133.0.123:15021: connect: connection refused 27m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Startup probe failed: Get "http://10.134.0.109:15021/healthz/ready": dial tcp 10.134.0.109:15021: connect: connection refused 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Startup probe error: Get "http://10.133.0.123:15021/healthz/ready": dial tcp 10.133.0.123:15021: connect: connection refused... 27m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Startup probe error: Get "http://10.134.0.109:15021/healthz/ready": dial tcp 10.134.0.109:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning Unhealthy pod/payload-processing-e2e-worker-w0-6a0a09-7c97c694d5-7d8lh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning ProbeError pod/e2e-api-b-ff338b-openshift-default-84498fd4f9-5lws9 Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-6a0a09-5974895ff9-q722b Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q 27m Normal Started pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Container started 27m Normal SuccessfulCreate replicaset/e2e-mt-422c1029-openshift-default-5786fb5789 Created pod: e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-422c1029-openshift-default New size: 10; reason: 27m Normal ScalingReplicaSet deployment/e2e-mt-422c1029-openshift-default Scaled up replica set e2e-mt-422c1029-openshift-default-5786fb5789 from 8 to 10 27m Normal Pulled pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Container created 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Add eth0 [10.134.0.110/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Add eth0 [10.133.0.124/23] from ovn-kubernetes 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Container started 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Container created 27m Normal Started pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Container started 27m Normal Created pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Container created 27m Normal Pulled pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Startup probe error: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Startup probe error: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Startup probe failed: Get "http://10.132.0.217:15021/healthz/ready": dial tcp 10.132.0.217:15021: connect: connection refused 27m Warning ProbeError pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Startup probe error: Get "http://10.132.0.217:15021/healthz/ready": dial tcp 10.132.0.217:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Startup probe failed: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused 27m Warning Unhealthy pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Startup probe failed: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused 27m Normal Killing pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Stopping container payload-pre-processing 27m Normal AddedLabel gateway/e2e-api-b-d59c58 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-d59c58 27m Normal ScalingReplicaSet deployment/e2e-api-b-d59c58-openshift-default Scaled up replica set e2e-api-b-d59c58-openshift-default-79b4dc54b7 from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-api-b-d59c58-openshift-default-79b4dc54b7 Created pod: e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb 27m Normal AddedInterface pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Add eth0 [10.134.0.111/23] from ovn-kubernetes 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-d67bf9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-d67bf9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-9cff60-6867b4bb8c-njlgq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-processing-e2e-worker-w1-9cff60-796b546677-8sxzm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-d59c58 Scaled up replica set payload-processing-e2e-api-b-d59c58-5b46f96fb9 from 0 to 1 27m Warning FailedCreate replicaset/payload-processing-e2e-api-b-d59c58-5b46f96fb9 Error creating: pods "payload-processing-e2e-api-b-d59c58-5b46f96fb9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-d59c58: serviceaccount "payload-processing-e2e-api-b-d59c58" not found 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-d59c58 Scaled up replica set payload-pre-processing-e2e-api-b-d59c58-7678d58759 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-d59c58-7678d58759 Created pod: payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-d59c58-5b46f96fb9 Created pod: payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-d59c58-7678d58759 Error creating: pods "payload-pre-processing-e2e-api-b-d59c58-7678d58759-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-d59c58: serviceaccount "payload-processing-e2e-api-b-d59c58" not found 27m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Add eth0 [10.134.0.113/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Container started 27m Normal AddedInterface pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Add eth0 [10.134.0.112/23] from ovn-kubernetes 27m Normal Pulled pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Pulled pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Container created 27m Normal Created pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Container created 27m Normal Started pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Container started 26m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-s4qh5 Stopping container istio-proxy 26m Warning ProbeError pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Readiness probe error: HTTP probe failed with statuscode: 503... 26m Warning Unhealthy pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf Readiness probe failed: HTTP probe failed with statuscode: 503 26m Normal Killing pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Stopping container payload-processing 26m Normal Killing pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Stopping container payload-pre-processing 26m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-njpdx Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-zct7m Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-74nsh Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-vr9nm Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-ghjrs Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-jsm7q Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-99f4v Stopping container istio-proxy 26m Normal Killing pod/e2e-mt-422c1029-openshift-default-5786fb5789-4mdtl Stopping container istio-proxy 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Readiness probe error: Get "http://10.134.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Readiness probe error: Get "http://10.133.0.123:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p Readiness probe failed: Get "http://10.134.0.109:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Readiness probe error: Get "http://10.133.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-2wc6x Readiness probe failed: Get "http://10.133.0.123:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-5l25z Readiness probe failed: Get "http://10.133.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-d8837a-67577b7475-gqcjk Readiness probe failed: timeout: failed to connect service "10.132.0.66:50052" within 1s: context deadline exceeded 26m Warning Unhealthy pod/payload-processing-e2e-worker-w2-d8837a-6bdbb559b9-lhhrc Readiness probe failed: timeout: failed to connect service "10.132.0.67:50052" within 1s: context deadline exceeded 25m Warning ProbeError pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Readiness probe error: Get "http://10.134.0.111:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning Unhealthy pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Readiness probe failed: Get "http://10.134.0.111:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Normal Started pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Container started 25m Normal Created pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Container created 25m Normal Pulled pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning ProbeError pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Startup probe error: Get "http://10.134.0.111:15021/healthz/ready": dial tcp 10.134.0.111:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Startup probe failed: Get "http://10.134.0.111:15021/healthz/ready": dial tcp 10.134.0.111:15021: connect: connection refused 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-d59c58-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-d59c58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Normal Killing pod/payload-processing-e2e-api-a-d67bf9-6fb7f9c547-mpdtz Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-api-a-d67bf9-6b574779b7-k6tw2 Stopping container payload-pre-processing 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-d59c58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-d59c58-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Stopping container istio-proxy 25m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Readiness probe failed: HTTP probe failed with statuscode: 503 25m Normal Killing pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-f7nrw Stopping container istio-proxy 25m Warning ProbeError pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Readiness probe error: HTTP probe failed with statuscode: 503... 25m Warning Unhealthy pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh Readiness probe failed: HTTP probe failed with statuscode: 503 25m Normal Killing pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Stopping container payload-processing 24m Normal Killing pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Stopping container payload-processing 24m Normal Killing pod/e2e-api-a-d67bf9-openshift-default-7585d64b69-wqqxc Stopping container istio-proxy 24m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-2182a5-97598c48c-zc65f Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning Unhealthy pod/payload-processing-e2e-worker-w6-2182a5-597f4f4f45-npxnv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal Killing pod/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g Stopping container payload-processing 24m Warning Unhealthy pod/payload-processing-e2e-worker-w5-3783c2-846f9695b9-lg95g Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 24m Normal ScalingReplicaSet deployment/e2e-worker-w5-3783c2-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w5-3783c2-openshift-default-7c54dd88dc from 3 to 2 24m Normal SuccessfulDelete replicaset/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc Deleted pod: e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-3783c2-openshift-default New size: 2; reason: All metrics below target 24m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-557sp Stopping container istio-proxy 24m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Stopping container istio-proxy 24m Normal Killing pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Stopping container istio-proxy 24m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-f85vf Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 Readiness probe failed: HTTP probe failed with statuscode: 503 24m Normal Killing pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb Stopping container istio-proxy 24m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-qn9j6 Stopping container istio-proxy 24m Normal Killing pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-84k9r Stopping container istio-proxy 24m Normal Killing pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Stopping container payload-pre-processing 24m Normal Killing pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Stopping container payload-processing 23m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-8ca52a-5bc9ff8bff-kfqgd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Warning Unhealthy pod/payload-processing-e2e-worker-w3-8ca52a-7ffc84dcfc-p6zhr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 23m Normal Killing pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Stopping container istio-proxy 23m Normal Killing pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Stopping container istio-proxy 23m Warning Unhealthy pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning ProbeError pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-sgljn Readiness probe error: HTTP probe failed with statuscode: 503... 23m Warning ProbeError pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Readiness probe error: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 23m Warning Unhealthy pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-jbmnz Readiness probe failed: Get "http://10.133.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-6dg8m 21m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 21m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-6dg8m Stopping container discovery 21m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 20m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Readiness probe error: Get "http://10.134.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh Readiness probe failed: Get "http://10.134.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Readiness probe error: Get "http://10.132.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-nqjll Readiness probe failed: Get "http://10.132.0.224:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 20m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Readiness probe error: Get "http://10.133.0.120:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 20m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-whxxj Readiness probe failed: Get "http://10.133.0.120:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Readiness probe failed: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 17m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-vmh57 Readiness probe error: Get "http://10.133.0.32:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Readiness probe error: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-zwvws Readiness probe failed: Get "http://10.134.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 15m Normal SuccessfulCreate replicaset/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f Created pod: e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw 15m Normal ScalingReplicaSet deployment/e2e-worker-single-main-ca3b17-openshift-default Scaled up replica set e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f from 0 to 1 15m Normal AddedLabel gateway/e2e-worker-single-main-ca3b17 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-ca3b17 15m Normal AddedInterface pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Add eth0 [10.132.1.2/23] from ovn-kubernetes 15m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-ca3b17 Scaled up replica set payload-processing-e2e-worker-single-main-ca3b17-78948fcfcc from 0 to 1 15m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-ca3b17-56755fffc5 Error creating: pods "payload-pre-processing-e2e-worker-single-main-ca3b17-56755fffc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17: serviceaccount "payload-processing-e2e-worker-single-main-ca3b17" not found 15m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcc Created pod: payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk 15m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcc Error creating: pods "payload-processing-e2e-worker-single-main-ca3b17-78948fcfcc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ca3b17: serviceaccount "payload-processing-e2e-worker-single-main-ca3b17" not found 15m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-ca3b17 Scaled up replica set payload-pre-processing-e2e-worker-single-main-ca3b17-56755fffc5 from 0 to 1 15m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-ca3b17-56755fffc5 Created pod: payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 15m Normal Started pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Container started 15m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Add eth0 [10.133.0.138/23] from ovn-kubernetes 15m Normal Started pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Container started 15m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Normal Created pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Container created 15m Normal Created pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Container created 15m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Add eth0 [10.132.1.3/23] from ovn-kubernetes 15m Normal Pulled pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 15m Normal Pulled pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 15m Normal Created pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Container created 15m Normal Started pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Container started 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ca3b17-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ca3b17-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Warning ProbeError pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Startup probe error: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused... 15m Warning Unhealthy pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Startup probe failed: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused 14m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ca3b17-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ca3b17-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Normal Killing pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Stopping container payload-processing 14m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Stopping container payload-pre-processing 13m Normal Killing pod/e2e-worker-single-main-ca3b17-openshift-default-5ddf8cf64f8rtcw Stopping container istio-proxy 13m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-ca3b17-78948fcfcltlzk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 13m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-ca3b17-56755r9vx2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 10m Normal AddedLabel gateway/e2e-worker-single-main-ef21ee Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-ef21ee 10m Normal SuccessfulCreate replicaset/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5 Created pod: e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 10m Normal AddedInterface pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Add eth0 [10.132.1.10/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/e2e-worker-single-main-ef21ee-openshift-default Scaled up replica set e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-ef21ee Scaled up replica set payload-processing-e2e-worker-single-main-ef21ee-7868cb5dbc from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bc6b787 Created pod: payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bc6b787 Error creating: pods "payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bc6b787-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee: serviceaccount "payload-processing-e2e-worker-single-main-ef21ee" not found 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-ef21ee Scaled up replica set payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bc6b787 from 0 to 1 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-ef21ee-7868cb5dbc Error creating: pods "payload-processing-e2e-worker-single-main-ef21ee-7868cb5dbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-ef21ee: serviceaccount "payload-processing-e2e-worker-single-main-ef21ee" not found 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-ef21ee-7868cb5dbc Created pod: payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc 10m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Add eth0 [10.133.0.140/23] from ovn-kubernetes 10m Normal Pulled pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Created pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Container created 10m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Container started 10m Normal Created pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Container created 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Add eth0 [10.132.1.11/23] from ovn-kubernetes 10m Normal Started pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Container started 10m Normal Started pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Container started 10m Normal Created pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Container created 10m Normal Pulled pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Warning Unhealthy pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Startup probe failed: Get "http://10.132.1.10:15021/healthz/ready": dial tcp 10.132.1.10:15021: connect: connection refused 10m Warning ProbeError pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Startup probe error: Get "http://10.132.1.10:15021/healthz/ready": dial tcp 10.132.1.10:15021: connect: connection refused... 9m58s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-ef21ee-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m58s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-ef21ee-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m18s Normal Killing pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Stopping container payload-processing 9m18s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Stopping container payload-pre-processing 9m5s Normal Killing pod/e2e-worker-single-main-ef21ee-openshift-default-6bbdc7b4d5rftk4 Stopping container istio-proxy 8m58s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-ef21ee-6f8bcpsxmq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 8m57s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-ef21ee-7868cb5db2tfnc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m22s Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Readiness probe failed: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m22s Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-t54jj Readiness probe error: Get "http://10.132.0.165:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m26s Normal ScalingReplicaSet deployment/e2e-ait-7e1fe5c2-openshift-default Scaled up replica set e2e-ait-7e1fe5c2-openshift-default-b5c9649c7 from 0 to 1 6m26s Normal SuccessfulCreate replicaset/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7 Created pod: e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz 6m26s Normal AddedLabel gateway/e2e-ait-7e1fe5c2 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-7e1fe5c2 6m25s Normal AddedInterface pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Add eth0 [10.132.1.19/23] from ovn-kubernetes 6m18s Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-7e1fe5c2 Scaled up replica set payload-processing-e2e-ait-7e1fe5c2-67869748c from 0 to 1 6m18s Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-7e1fe5c2-67869748c Created pod: payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr 6m18s Warning FailedCreate replicaset/payload-processing-e2e-ait-7e1fe5c2-67869748c Error creating: pods "payload-processing-e2e-ait-7e1fe5c2-67869748c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-7e1fe5c2: serviceaccount "payload-processing-e2e-ait-7e1fe5c2" not found 6m18s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-7e1fe5c2 Scaled up replica set payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5 from 0 to 1 6m18s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5 Created pod: payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 6m18s Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5 Error creating: pods "payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-7e1fe5c2: serviceaccount "payload-processing-e2e-ait-7e1fe5c2" not found 6m17s Normal Pulled pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m17s Normal AddedInterface pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Add eth0 [10.132.1.20/23] from ovn-kubernetes 6m17s Normal AddedInterface pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Add eth0 [10.132.1.21/23] from ovn-kubernetes 6m17s Normal Created pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Container created 6m17s Normal Started pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Container started 6m17s Normal Started pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Container started 6m17s Normal Created pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Container created 6m17s Normal Pulled pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 6m10s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-7e1fe5c2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 6m10s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-7e1fe5c2-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m59s Normal Created pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Container created 5m59s Normal Started pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Container started 5m59s Normal Pulled pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m53s Warning Unhealthy pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Startup probe failed: Get "http://10.132.1.19:15021/healthz/ready": dial tcp 10.132.1.19:15021: connect: connection refused 5m53s Warning ProbeError pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Startup probe error: Get "http://10.132.1.19:15021/healthz/ready": dial tcp 10.132.1.19:15021: connect: connection refused... 5m35s Normal Killing pod/payload-processing-e2e-ait-7e1fe5c2-67869748c-bzvtr Stopping container payload-processing 5m34s Normal Killing pod/payload-pre-processing-e2e-ait-7e1fe5c2-748d6bdc5-9s229 Stopping container payload-pre-processing 5m25s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-7e1fe5c2-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m25s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-7e1fe5c2-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 5m19s Normal Killing pod/e2e-ait-7e1fe5c2-openshift-default-b5c9649c7-7s2qz Stopping container istio-proxy 5m17s Normal ScalingReplicaSet deployment/e2e-authcfg-http-40b9a8-openshift-default Scaled up replica set e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf from 0 to 1 5m17s Normal SuccessfulCreate replicaset/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf Created pod: e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l 5m17s Normal AddedLabel gateway/e2e-authcfg-http-40b9a8 Added label istio.io/rev=openshift-gateway to gateway e2e-authcfg-http-40b9a8 5m16s Normal AddedInterface pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Add eth0 [10.132.1.23/23] from ovn-kubernetes 5m9s Normal SuccessfulCreate replicaset/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587 Created pod: payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw 5m9s Normal ScalingReplicaSet deployment/payload-processing-e2e-authcfg-http-40b9a8 Scaled up replica set payload-processing-e2e-authcfg-http-40b9a8-687b6b5587 from 0 to 1 5m9s Warning FailedCreate replicaset/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5 Error creating: pods "payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8: serviceaccount "payload-processing-e2e-authcfg-http-40b9a8" not found 5m9s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5 Created pod: payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt 5m9s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-authcfg-http-40b9a8 Scaled up replica set payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5 from 0 to 1 5m9s Warning FailedCreate replicaset/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587 Error creating: pods "payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-40b9a8: serviceaccount "payload-processing-e2e-authcfg-http-40b9a8" not found 5m8s Normal Started pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Container started 5m8s Normal AddedInterface pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Add eth0 [10.132.1.24/23] from ovn-kubernetes 5m8s Normal Pulled pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m8s Normal Created pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Container created 5m8s Normal Started pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Container started 5m8s Normal Created pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Container created 5m8s Normal Pulled pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m8s Normal AddedInterface pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Add eth0 [10.132.1.25/23] from ovn-kubernetes 4m48s Normal Started pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Container started 4m48s Normal Created pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Container created 4m48s Normal Pulled pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m47s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-40b9a8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m47s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-40b9a8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m43s Warning Unhealthy pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Startup probe failed: Get "http://10.132.1.23:15021/healthz/ready": dial tcp 10.132.1.23:15021: connect: connection refused 4m43s Warning ProbeError pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Startup probe error: Get "http://10.132.1.23:15021/healthz/ready": dial tcp 10.132.1.23:15021: connect: connection refused... 4m1s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-40b9a8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m1s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-40b9a8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 3m17s Normal Killing pod/payload-processing-e2e-authcfg-http-40b9a8-687b6b5587-6sshw Stopping container payload-processing 3m16s Normal Killing pod/payload-pre-processing-e2e-authcfg-http-40b9a8-6f4c58cdf5-254xt Stopping container payload-pre-processing 3m Normal Killing pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Stopping container istio-proxy 2m57s Warning Unhealthy pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Readiness probe failed: HTTP probe failed with statuscode: 503 2m57s Warning ProbeError pod/e2e-authcfg-http-40b9a8-openshift-default-5f98fb99bf-pzj4l Readiness probe error: HTTP probe failed with statuscode: 503... 106s Warning Unhealthy pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Readiness probe failed: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 106s Warning ProbeError pod/maas-default-gateway-openshift-default-67bf986f99-kkskg Readiness probe error: Get "http://10.132.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...