=== events in openshift-ingress (2026-10-31T18:23:19Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 57m Normal Scheduled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Successfully assigned openshift-ingress/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws to ip-10-0-140-192.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 to ip-10-0-138-123.ec2.internal 57m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z to ip-10-0-140-192.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k to ip-10-0-138-123.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j to ip-10-0-140-192.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Successfully assigned openshift-ingress/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf to ip-10-0-140-192.ec2.internal 33m Normal Scheduled pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Successfully assigned openshift-ingress/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 to ip-10-0-140-192.ec2.internal 10m Normal Scheduled pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Successfully assigned openshift-ingress/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx to ip-10-0-142-138.ec2.internal 53m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 to ip-10-0-142-138.ec2.internal 49m Normal Scheduled pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Successfully assigned openshift-ingress/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 to ip-10-0-140-192.ec2.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl to ip-10-0-142-138.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb to ip-10-0-138-123.ec2.internal 45m Normal Scheduled pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Successfully assigned openshift-ingress/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb to ip-10-0-142-138.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 to ip-10-0-138-123.ec2.internal 37m Normal Scheduled pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Successfully assigned openshift-ingress/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh to ip-10-0-138-123.ec2.internal 14m Normal Scheduled pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Successfully assigned openshift-ingress/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w to ip-10-0-140-192.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg to ip-10-0-138-123.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-7b98bb678c-8l7tw Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-8l7tw to ip-10-0-142-138.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 to ip-10-0-140-192.ec2.internal 58m Normal Scheduled pod/payload-processing-55fbbbf9dc-w7cc7 Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-w7cc7 to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq to ip-10-0-138-123.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Successfully assigned openshift-ingress/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz to ip-10-0-140-192.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Successfully assigned openshift-ingress/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb to ip-10-0-140-192.ec2.internal 48m Normal Scheduled pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Successfully assigned openshift-ingress/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 to ip-10-0-140-192.ec2.internal 18m Normal Scheduled pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Successfully assigned openshift-ingress/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 to ip-10-0-142-138.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 to ip-10-0-138-123.ec2.internal 46m Normal Scheduled pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Successfully assigned openshift-ingress/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb to ip-10-0-140-192.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Successfully assigned openshift-ingress/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb to ip-10-0-140-192.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Successfully assigned openshift-ingress/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk to ip-10-0-138-123.ec2.internal 64m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g to ip-10-0-142-138.ec2.internal 48m Normal Scheduled pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Successfully assigned openshift-ingress/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 to ip-10-0-140-192.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j to ip-10-0-138-123.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m to ip-10-0-142-138.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Successfully assigned openshift-ingress/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 to ip-10-0-138-123.ec2.internal 37m Normal Scheduled pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Successfully assigned openshift-ingress/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w to ip-10-0-140-192.ec2.internal 10m Normal Scheduled pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Successfully assigned openshift-ingress/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf to ip-10-0-140-192.ec2.internal 56m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 to ip-10-0-138-123.ec2.internal 9m43s Normal Scheduled pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Successfully assigned openshift-ingress/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Successfully assigned openshift-ingress/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 to ip-10-0-140-192.ec2.internal 53m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs to ip-10-0-140-192.ec2.internal 37m Normal Scheduled pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg to ip-10-0-142-138.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 to ip-10-0-140-192.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 to ip-10-0-138-123.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m to ip-10-0-140-192.ec2.internal 33m Normal Scheduled pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Successfully assigned openshift-ingress/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m to ip-10-0-142-138.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv to ip-10-0-142-138.ec2.internal 71m Normal Scheduled pod/router-default-675867b879-l24qc Successfully assigned openshift-ingress/router-default-675867b879-l24qc to ip-10-0-142-138.ec2.internal 71m Warning FailedScheduling pod/router-default-675867b879-l24qc no nodes available to schedule pods 49m Normal Scheduled pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Successfully assigned openshift-ingress/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs to ip-10-0-138-123.ec2.internal 52m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m to ip-10-0-142-138.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn to ip-10-0-138-123.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 to ip-10-0-142-138.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk to ip-10-0-138-123.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r to ip-10-0-140-192.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh to ip-10-0-140-192.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb to ip-10-0-140-192.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md to ip-10-0-138-123.ec2.internal 41m Normal Scheduled pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Successfully assigned openshift-ingress/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb to ip-10-0-138-123.ec2.internal 45m Normal Scheduled pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Successfully assigned openshift-ingress/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw to ip-10-0-138-123.ec2.internal 42m Normal Scheduled pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Successfully assigned openshift-ingress/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt to ip-10-0-140-192.ec2.internal 45m Normal Scheduled pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Successfully assigned openshift-ingress/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx to ip-10-0-140-192.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n to ip-10-0-140-192.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 to ip-10-0-140-192.ec2.internal 44m Normal Scheduled pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Successfully assigned openshift-ingress/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 to ip-10-0-138-123.ec2.internal 77m Warning FailedScheduling pod/router-default-675867b879-l24qc no nodes available to schedule pods 52m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Successfully assigned openshift-ingress/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 to ip-10-0-142-138.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg to ip-10-0-140-192.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k to ip-10-0-142-138.ec2.internal 47m Normal Scheduled pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Successfully assigned openshift-ingress/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl to ip-10-0-140-192.ec2.internal 44m Normal Scheduled pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Successfully assigned openshift-ingress/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k to ip-10-0-138-123.ec2.internal 35m Normal Scheduled pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 to ip-10-0-140-192.ec2.internal 53m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 to ip-10-0-142-138.ec2.internal 37m Normal Scheduled pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd to ip-10-0-140-192.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Successfully assigned openshift-ingress/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv to ip-10-0-138-123.ec2.internal 53m Normal Scheduled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Successfully assigned openshift-ingress/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv to ip-10-0-138-123.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Successfully assigned openshift-ingress/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 to ip-10-0-140-192.ec2.internal 50m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg to ip-10-0-140-192.ec2.internal 45m Normal Scheduled pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Successfully assigned openshift-ingress/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd to ip-10-0-142-138.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Successfully assigned openshift-ingress/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw to ip-10-0-138-123.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 to ip-10-0-140-192.ec2.internal 43m Normal Scheduled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Successfully assigned openshift-ingress/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm to ip-10-0-140-192.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 to ip-10-0-138-123.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg to ip-10-0-140-192.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Successfully assigned openshift-ingress/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk to ip-10-0-138-123.ec2.internal 52m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 to ip-10-0-142-138.ec2.internal 49m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 to ip-10-0-138-123.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf to ip-10-0-142-138.ec2.internal 42m Normal Scheduled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Successfully assigned openshift-ingress/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp to ip-10-0-142-138.ec2.internal 54m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs to ip-10-0-140-192.ec2.internal 55m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x to ip-10-0-140-192.ec2.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv to ip-10-0-142-138.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v to ip-10-0-140-192.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Successfully assigned openshift-ingress/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 to ip-10-0-142-138.ec2.internal 42m Normal Scheduled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Successfully assigned openshift-ingress/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 to ip-10-0-138-123.ec2.internal 52m Normal Scheduled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Successfully assigned openshift-ingress/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd to ip-10-0-140-192.ec2.internal 54m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv to ip-10-0-138-123.ec2.internal 49m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Successfully assigned openshift-ingress/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs to ip-10-0-140-192.ec2.internal 54m Normal Scheduled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Successfully assigned openshift-ingress/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t to ip-10-0-138-123.ec2.internal 49m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 to ip-10-0-138-123.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv to ip-10-0-138-123.ec2.internal 42m Normal Scheduled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Successfully assigned openshift-ingress/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr to ip-10-0-142-138.ec2.internal 40m Normal Scheduled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Successfully assigned openshift-ingress/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj to ip-10-0-138-123.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f to ip-10-0-138-123.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np to ip-10-0-140-192.ec2.internal 50m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj to ip-10-0-138-123.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Successfully assigned openshift-ingress/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Successfully assigned openshift-ingress/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz to ip-10-0-138-123.ec2.internal 50m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv to ip-10-0-142-138.ec2.internal 56m Normal Scheduled pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n to ip-10-0-142-138.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 to ip-10-0-138-123.ec2.internal 64m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-m4552 to ip-10-0-142-138.ec2.internal 9m43s Normal Scheduled pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Successfully assigned openshift-ingress/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 to ip-10-0-138-123.ec2.internal 26m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-lshsj to ip-10-0-142-138.ec2.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 to ip-10-0-140-192.ec2.internal 64m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-lq4mh Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-lq4mh to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz to ip-10-0-138-123.ec2.internal 49m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf to ip-10-0-142-138.ec2.internal 43m Normal Scheduled pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Successfully assigned openshift-ingress/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Successfully assigned openshift-ingress/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct to ip-10-0-138-123.ec2.internal 53m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx to ip-10-0-140-192.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq to ip-10-0-138-123.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 to ip-10-0-140-192.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms to ip-10-0-138-123.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq to ip-10-0-140-192.ec2.internal 26m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-d8glx to ip-10-0-140-192.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm to ip-10-0-138-123.ec2.internal 42m Normal Scheduled pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Successfully assigned openshift-ingress/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 to ip-10-0-140-192.ec2.internal 61m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6w97r to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d76xz to ip-10-0-142-138.ec2.internal 51m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl to ip-10-0-140-192.ec2.internal 51m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr to ip-10-0-142-138.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj to ip-10-0-138-123.ec2.internal 57m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl to ip-10-0-140-192.ec2.internal 49m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lg2gf to ip-10-0-138-123.ec2.internal 43m Normal Scheduled pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Successfully assigned openshift-ingress/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j to ip-10-0-140-192.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s to ip-10-0-140-192.ec2.internal 40m Normal Scheduled pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Successfully assigned openshift-ingress/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x to ip-10-0-138-123.ec2.internal 51m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 to ip-10-0-138-123.ec2.internal 52m Normal Scheduled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Successfully assigned openshift-ingress/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf to ip-10-0-140-192.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc to ip-10-0-140-192.ec2.internal 62m Normal Scheduled pod/kube-auth-proxy-6bf694d79-5bsl4 Successfully assigned openshift-ingress/kube-auth-proxy-6bf694d79-5bsl4 to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk to ip-10-0-138-123.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk to ip-10-0-142-138.ec2.internal 35m Normal Scheduled pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Successfully assigned openshift-ingress/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t to ip-10-0-140-192.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw to ip-10-0-142-138.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm to ip-10-0-140-192.ec2.internal 62m Normal Scheduled pod/kube-auth-proxy-6bf694d79-z92tm Successfully assigned openshift-ingress/kube-auth-proxy-6bf694d79-z92tm to ip-10-0-138-123.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp to ip-10-0-138-123.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 to ip-10-0-138-123.ec2.internal 42m Normal Scheduled pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Successfully assigned openshift-ingress/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j to ip-10-0-138-123.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Successfully assigned openshift-ingress/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx to ip-10-0-140-192.ec2.internal 42m Normal Scheduled pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Successfully assigned openshift-ingress/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz to ip-10-0-138-123.ec2.internal 26m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-cf442 to ip-10-0-142-138.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g to ip-10-0-138-123.ec2.internal 26m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c to ip-10-0-140-192.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 to ip-10-0-138-123.ec2.internal 57m Normal Scheduled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Successfully assigned openshift-ingress/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w to ip-10-0-138-123.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k to ip-10-0-140-192.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 to ip-10-0-138-123.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc to ip-10-0-140-192.ec2.internal 35m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn to ip-10-0-140-192.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz to ip-10-0-138-123.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr to ip-10-0-140-192.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Successfully assigned openshift-ingress/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r to ip-10-0-138-123.ec2.internal 26m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-2msfs to ip-10-0-140-192.ec2.internal 9m49s Normal Scheduled pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Successfully assigned openshift-ingress/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv to ip-10-0-140-192.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Successfully assigned openshift-ingress/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s to ip-10-0-140-192.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Successfully assigned openshift-ingress/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh to ip-10-0-142-138.ec2.internal 47m Normal Scheduled pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Successfully assigned openshift-ingress/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr to ip-10-0-140-192.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml to ip-10-0-140-192.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml to ip-10-0-140-192.ec2.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp to ip-10-0-140-192.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx to ip-10-0-142-138.ec2.internal 62m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 to ip-10-0-142-138.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 to ip-10-0-138-123.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Successfully assigned openshift-ingress/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn to ip-10-0-140-192.ec2.internal 37m Normal Scheduled pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Successfully assigned openshift-ingress/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv to ip-10-0-140-192.ec2.internal 57m Normal Scheduled pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 to ip-10-0-142-138.ec2.internal 49m Normal Scheduled pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Successfully assigned openshift-ingress/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 to ip-10-0-138-123.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h to ip-10-0-140-192.ec2.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b to ip-10-0-142-138.ec2.internal 54m Normal Scheduled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Successfully assigned openshift-ingress/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 to ip-10-0-142-138.ec2.internal 54m Normal Scheduled pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k to ip-10-0-138-123.ec2.internal 78m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-675867b879 from 0 to 1 78m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 78m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 77m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 77m Normal SuccessfulCreate replicaset/router-default-675867b879 Created pod: router-default-675867b879-l24qc 71m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 71m Warning FailedMount pod/router-default-675867b879-l24qc MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 71m Warning FailedMount pod/router-default-675867b879-l24qc MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 70m Warning FailedMount pod/router-default-675867b879-l24qc MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 70m Warning FailedMount pod/router-default-675867b879-l24qc MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 69m Normal AddedInterface pod/router-default-675867b879-l24qc Add eth0 [10.132.0.12/23] from ovn-kubernetes 69m Normal Pulled pod/router-default-675867b879-l24qc Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 69m Normal Created pod/router-default-675867b879-l24qc Container created 69m Normal Started pod/router-default-675867b879-l24qc Container started 64m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 64m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-lq4mh 64m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 64m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-lq4mh Add eth0 [10.134.0.22/23] from ovn-kubernetes 64m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-lq4mh Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 64m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-lq4mh Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 9.25s (9.25s including waiting). Image size: 185476477 bytes. 64m Normal Created pod/istiod-openshift-gateway-7876bfcb74-lq4mh Container created 64m Normal Started pod/istiod-openshift-gateway-7876bfcb74-lq4mh Container started 64m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9 Created pod: openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g 64m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-lq4mh Readiness probe error: HTTP probe failed with statuscode: 503... 64m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 0 to 1 64m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-m4552 64m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 64m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-6c77f4c8c9 from 0 to 1 64m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-lq4mh Readiness probe failed: HTTP probe failed with statuscode: 503 64m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 64m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Add eth0 [10.132.0.41/23] from ovn-kubernetes 64m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Add eth0 [10.132.0.40/23] from ovn-kubernetes 64m Normal Pulling pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 64m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 64m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.033s (10.033s including waiting). Image size: 183555333 bytes. 64m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.677s (9.677s including waiting). Image size: 183555333 bytes. 64m Normal Started pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Container started 64m Normal Created pod/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-ndr2g Container created 63m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 63m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 63m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-6bf694d79 from 0 to 2 62m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 Add eth0 [10.132.0.48/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-5c79575b6f Created pod: data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 62m Normal AddedInterface pod/kube-auth-proxy-6bf694d79-5bsl4 Add eth0 [10.134.0.25/23] from ovn-kubernetes 62m Normal AddedInterface pod/kube-auth-proxy-6bf694d79-z92tm Add eth0 [10.133.0.21/23] from ovn-kubernetes 62m Normal SuccessfulCreate replicaset/kube-auth-proxy-6bf694d79 Created pod: kube-auth-proxy-6bf694d79-5bsl4 62m Normal SuccessfulCreate replicaset/kube-auth-proxy-6bf694d79 Created pod: kube-auth-proxy-6bf694d79-z92tm 62m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-5c79575b6f from 0 to 1 62m Normal Pulling pod/kube-auth-proxy-6bf694d79-5bsl4 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 62m Normal Pulling pod/kube-auth-proxy-6bf694d79-z92tm Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 62m Normal Pulled pod/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 62m Normal Started pod/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 Container started 62m Normal Created pod/data-science-gateway-data-science-gateway-class-5c79575b6f5qvf5 Container created 62m Normal Started pod/kube-auth-proxy-6bf694d79-5bsl4 Container started 62m Normal Created pod/kube-auth-proxy-6bf694d79-5bsl4 Container created 62m Normal Pulled pod/kube-auth-proxy-6bf694d79-5bsl4 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.625s (2.625s including waiting). Image size: 188904677 bytes. 62m Normal Created pod/kube-auth-proxy-6bf694d79-z92tm Container created 62m Normal Started pod/kube-auth-proxy-6bf694d79-z92tm Container started 62m Normal Pulled pod/kube-auth-proxy-6bf694d79-z92tm Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.732s (2.732s including waiting). Image size: 188904677 bytes. 61m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 61m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Add eth0 [10.134.0.31/23] from ovn-kubernetes 61m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 61m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 61m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Container created 61m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Container started 61m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-6w97r 61m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Readiness probe error: HTTP probe failed with statuscode: 503... 61m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-6w97r Readiness probe failed: HTTP probe failed with statuscode: 503 61m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-lq4mh Stopping container discovery 61m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-lq4mh 61m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 60m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Readiness probe failed: Get "http://10.132.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 60m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Readiness probe error: Get "http://10.132.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 60m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Container created 60m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Container started 60m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Startup probe failed: Get "http://10.132.0.40:15021/healthz/ready": dial tcp 10.132.0.40:15021: connect: connection refused 60m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Startup probe error: Get "http://10.132.0.40:15021/healthz/ready": dial tcp 10.132.0.40:15021: connect: connection refused... 59m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 59m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 1 to 4 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-79dq2 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-cvtfw 58m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-26sm5 58m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Add eth0 [10.133.0.27/23] from ovn-kubernetes 58m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Container started 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Add eth0 [10.132.0.58/23] from ovn-kubernetes 58m Normal Pulling pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 58m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Container created 58m Normal Pulling pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Add eth0 [10.134.0.41/23] from ovn-kubernetes 58m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.673s (3.673s including waiting). Image size: 183555333 bytes. 58m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.896s (2.896s including waiting). Image size: 183555333 bytes. 58m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Container started 58m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Container created 58m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Container created 58m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Container started 58m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Startup probe failed: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused 58m Normal SuccessfulCreate replicaset/payload-processing-55fbbbf9dc Created pod: payload-processing-55fbbbf9dc-w7cc7 58m Warning FailedCreate replicaset/payload-processing-55fbbbf9dc Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 58m Normal Pulling pod/payload-pre-processing-7b98bb678c-8l7tw Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 58m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-8l7tw 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Startup probe error: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused... 58m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 58m Normal AddedInterface pod/payload-processing-55fbbbf9dc-w7cc7 Add eth0 [10.132.0.61/23] from ovn-kubernetes 58m Normal Pulling pod/payload-processing-55fbbbf9dc-w7cc7 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-8l7tw Add eth0 [10.132.0.62/23] from ovn-kubernetes 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Startup probe failed: Get "http://10.133.0.27:15021/healthz/ready": dial tcp 10.133.0.27:15021: connect: connection refused 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Startup probe error: Get "http://10.133.0.27:15021/healthz/ready": dial tcp 10.133.0.27:15021: connect: connection refused... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Startup probe failed: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Startup probe error: Get "http://10.132.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 58m Normal Created pod/payload-processing-55fbbbf9dc-w7cc7 Container created 58m Normal Pulled pod/payload-pre-processing-7b98bb678c-8l7tw Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.567s (3.567s including waiting). Image size: 125353963 bytes. 58m Normal Started pod/payload-processing-55fbbbf9dc-w7cc7 Container started 58m Normal Started pod/payload-pre-processing-7b98bb678c-8l7tw Container started 58m Normal Pulled pod/payload-processing-55fbbbf9dc-w7cc7 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.609s (3.609s including waiting). Image size: 125353963 bytes. 58m Normal Created pod/payload-pre-processing-7b98bb678c-8l7tw Container created 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Startup probe error: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Startup probe failed: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 58m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-vvv67 58m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 4 to 5 58m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Container created 58m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Container started 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Add eth0 [10.134.0.42/23] from ovn-kubernetes 58m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Startup probe error: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Startup probe failed: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused 57m Normal AddedLabel gateway/e2e-worker-w0-e1a7f5 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-e1a7f5 57m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 0 to 1 57m Normal AddedLabel gateway/e2e-worker-w4-bb48ef Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-bb48ef 57m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 0 to 1 57m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 0 to 1 57m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd 57m Normal AddedLabel gateway/e2e-worker-w1-7c933c Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-7c933c 57m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt 57m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f 57m Normal AddedLabel gateway/e2e-worker-w6-5d73b9 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-5d73b9 57m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z 57m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled up replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 0 to 1 57m Normal AddedLabel gateway/e2e-worker-w3-159cb7 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-159cb7 57m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Add eth0 [10.133.0.28/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Add eth0 [10.134.0.45/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Add eth0 [10.134.0.44/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w 57m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Add eth0 [10.134.0.46/23] from ovn-kubernetes 57m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled up replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 0 to 1 57m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Add eth0 [10.134.0.43/23] from ovn-kubernetes 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-7c933c-5b84567d97 Error creating: pods "payload-processing-e2e-worker-w1-7c933c-5b84567d97-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-7c933c: serviceaccount "payload-processing-e2e-worker-w1-7c933c" not found 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-7c933c Scaled up replica set payload-processing-e2e-worker-w1-7c933c-5b84567d97 from 0 to 1 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7 Error creating: pods "payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-7c933c: serviceaccount "payload-processing-e2e-worker-w1-7c933c" not found 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-7c933c Scaled up replica set payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7 from 0 to 1 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7 Created pod: payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-7c933c-5b84567d97 Created pod: payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 57m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Add eth0 [10.132.0.65/23] from ovn-kubernetes 57m Normal Started pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Container started 57m Normal Pulled pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal Created pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Container created 57m Normal Created pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Container created 57m Normal Started pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Container started 57m Normal AddedInterface pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Add eth0 [10.132.0.66/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff Created pod: payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-e1a7f5 Scaled up replica set payload-processing-e2e-worker-w0-e1a7f5-7859859c4f from 0 to 1 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff Error creating: pods "payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-e1a7f5: serviceaccount "payload-processing-e2e-worker-w0-e1a7f5" not found 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f Created pod: payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-e1a7f5 Scaled up replica set payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff from 0 to 1 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f Error creating: pods "payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-e1a7f5: serviceaccount "payload-processing-e2e-worker-w0-e1a7f5" not found 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-5d73b9 Scaled up replica set payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667 from 0 to 1 57m Normal AddedInterface pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Add eth0 [10.132.0.69/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667 Created pod: payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 57m Normal Started pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Container started 57m Normal Pulled pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-5d73b9 Scaled up replica set payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7 from 0 to 1 57m Normal Started pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Container started 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667 Error creating: pods "payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-5d73b9: serviceaccount "payload-processing-e2e-worker-w6-5d73b9" not found 57m Normal Created pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Container created 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7 Error creating: pods "payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-5d73b9: serviceaccount "payload-processing-e2e-worker-w6-5d73b9" not found 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7 Created pod: payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv 57m Normal Created pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Container created 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Add eth0 [10.132.0.68/23] from ovn-kubernetes 57m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal Pulled pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f Created pod: payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f Error creating: pods "payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-bb48ef: serviceaccount "payload-processing-e2e-worker-w4-bb48ef" not found 57m Normal AddedInterface pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Add eth0 [10.132.0.71/23] from ovn-kubernetes 57m Normal Started pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Container started 57m Normal Created pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Container created 57m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 0 to 1 57m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw 57m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled up replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 0 to 1 57m Normal SuccessfulCreate replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Created pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws 57m Normal Started pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Container started 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669 Error creating: pods "payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-bb48ef: serviceaccount "payload-processing-e2e-worker-w4-bb48ef" not found 57m Normal Created pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Container created 57m Normal AddedLabel gateway/e2e-worker-w5-e2e897 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-e2e897 57m Normal AddedLabel gateway/e2e-worker-w2-1fb1c6 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-1fb1c6 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-bb48ef Scaled up replica set payload-processing-e2e-worker-w4-bb48ef-7f4f99b669 from 0 to 1 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Add eth0 [10.132.0.70/23] from ovn-kubernetes 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669 Created pod: payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-bb48ef Scaled up replica set payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f from 0 to 1 57m Normal Started pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Container started 57m Normal AddedInterface pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Add eth0 [10.132.0.73/23] from ovn-kubernetes 57m Normal AddedInterface pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Add eth0 [10.134.0.47/23] from ovn-kubernetes 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Add eth0 [10.132.0.72/23] from ovn-kubernetes 57m Normal Started pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Container started 57m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Add eth0 [10.133.0.29/23] from ovn-kubernetes 57m Normal Created pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Container created 57m Normal Created pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Container created 57m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal Pulled pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-1fb1c6 Scaled up replica set payload-processing-e2e-worker-w2-1fb1c6-69b567b565 from 0 to 1 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5 Created pod: payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl 57m Normal AddedInterface pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Add eth0 [10.134.0.48/23] from ovn-kubernetes 57m Normal Pulling pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 57m Normal Pulling pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Add eth0 [10.134.0.49/23] from ovn-kubernetes 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-1fb1c6 Scaled up replica set payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5 from 0 to 1 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5 Error creating: pods "payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-1fb1c6: serviceaccount "payload-processing-e2e-worker-w2-1fb1c6" not found 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-1fb1c6-69b567b565 Created pod: payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-1fb1c6-69b567b565 Error creating: pods "payload-processing-e2e-worker-w2-1fb1c6-69b567b565-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-1fb1c6: serviceaccount "payload-processing-e2e-worker-w2-1fb1c6" not found 57m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Readiness probe error: Get "http://10.134.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 57m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-159cb7 Scaled up replica set payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79 from 0 to 1 57m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79 Error creating: pods "payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-159cb7: serviceaccount "payload-processing-e2e-worker-w3-159cb7" not found 57m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-159cb7-cc54b644c Error creating: pods "payload-processing-e2e-worker-w3-159cb7-cc54b644c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-159cb7: serviceaccount "payload-processing-e2e-worker-w3-159cb7" not found 57m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-159cb7-cc54b644c Created pod: payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn 57m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-159cb7 Scaled up replica set payload-processing-e2e-worker-w3-159cb7-cc54b644c from 0 to 1 57m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Readiness probe failed: Get "http://10.134.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 57m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79 Created pod: payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv 56m Normal AddedInterface pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Add eth0 [10.134.0.50/23] from ovn-kubernetes 56m Normal Pulling pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Add eth0 [10.133.0.30/23] from ovn-kubernetes 56m Normal Pulling pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 56m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Container created 56m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Container started 56m Normal Created pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Container created 56m Normal Created pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Container created 56m Normal Created pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Container created 56m Normal Started pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Container started 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.42s (2.42s including waiting). Image size: 125353963 bytes. 56m Normal Started pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Container started 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 5.162s (5.162s including waiting). Image size: 125353963 bytes. 56m Normal Pulled pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 5.123s (5.123s including waiting). Image size: 125353963 bytes. 56m Normal Started pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Container started 56m Normal Started pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Container started 56m Normal Created pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Container created 56m Normal Pulled pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.721s (2.721s including waiting). Image size: 125353963 bytes. 56m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-e2e897-75cbd44db9 Created pod: payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt 56m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc Error creating: pods "payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-e2e897: serviceaccount "payload-processing-e2e-worker-w5-e2e897" not found 56m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc Created pod: payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 56m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-e2e897-75cbd44db9 Error creating: pods "payload-processing-e2e-worker-w5-e2e897-75cbd44db9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-e2e897: serviceaccount "payload-processing-e2e-worker-w5-e2e897" not found 56m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-e2e897 Scaled up replica set payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc from 0 to 1 56m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-e2e897 Scaled up replica set payload-processing-e2e-worker-w5-e2e897-75cbd44db9 from 0 to 1 56m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Container started 56m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal Created pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Container created 56m Normal Pulled pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Add eth0 [10.132.0.77/23] from ovn-kubernetes 56m Normal Started pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Container started 56m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Container created 56m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Add eth0 [10.133.0.31/23] from ovn-kubernetes 56m Normal Started pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Container started 56m Normal Created pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Container created 56m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Container started 56m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Startup probe error: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused... 56m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Container created 56m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Container started 56m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Container created 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Startup probe failed: Get "http://10.133.0.28:15021/healthz/ready": dial tcp 10.133.0.28:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Startup probe error: Get "http://10.133.0.28:15021/healthz/ready": dial tcp 10.133.0.28:15021: connect: connection refused... 56m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Startup probe error: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Startup probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused 56m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Container created 56m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Container started 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Startup probe error: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Startup probe failed: Get "http://10.133.0.29:15021/healthz/ready": dial tcp 10.133.0.29:15021: connect: connection refused 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Container started 56m Normal Pulled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Container created 56m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Container created 56m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Container started 56m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Startup probe error: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Startup probe failed: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Startup probe error: Get "http://10.134.0.47:15021/healthz/ready": dial tcp 10.134.0.47:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 1 to 2 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 56m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Container started 56m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Container created 56m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Add eth0 [10.133.0.39/23] from ovn-kubernetes 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 55m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused... 55m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 55m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 1 to 2 55m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Container started 55m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Container created 55m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Add eth0 [10.134.0.57/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 55m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused... 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled up replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 1 to 2 54m Normal SuccessfulCreate replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Created pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t 54m Normal Created pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Container created 54m Normal Pulled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Add eth0 [10.133.0.44/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Container started 54m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 1 to 2 54m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 54m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Container started 54m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Container created 54m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Add eth0 [10.132.0.80/23] from ovn-kubernetes 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Container started 54m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Container created 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled up replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 1 to 2 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Add eth0 [10.133.0.45/23] from ovn-kubernetes 54m Warning Unhealthy pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Startup probe error: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused... 54m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 2 to 3 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 1 to 2 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 54m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Container created 54m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Add eth0 [10.132.0.81/23] from ovn-kubernetes 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 54m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Container started 54m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Add eth0 [10.133.0.47/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Container created 54m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Container started 54m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 54m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Add eth0 [10.134.0.59/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-7gt6k 54m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-rdnfs 54m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-pfbqv 54m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Add eth0 [10.133.0.55/23] from ovn-kubernetes 54m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Add eth0 [10.133.0.56/23] from ovn-kubernetes 54m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 5 to 8 53m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Container created 53m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Container started 53m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Container created 53m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Container started 53m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Container started 53m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Container created 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused 53m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused... 53m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Startup probe failed: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused 53m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Startup probe error: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused... 53m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-xvb52 53m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 (combined from similar events): Created pod: maas-default-gateway-openshift-default-9dccdc9b7-h4wkx 53m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 8 to 10 53m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Container started 53m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Container created 53m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Add eth0 [10.134.0.61/23] from ovn-kubernetes 53m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Add eth0 [10.132.0.83/23] from ovn-kubernetes 53m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Container created 53m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Container started 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 53m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 53m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 53m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 53m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled up replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 2 to 4 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np 53m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 2 to 3 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs 53m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 53m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled up replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 1 to 2 53m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Container started 53m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Add eth0 [10.134.0.62/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Add eth0 [10.132.0.85/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Container created 53m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Container created 53m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Container started 53m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Container started 53m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Container created 53m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Add eth0 [10.132.0.86/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Add eth0 [10.134.0.63/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Container started 53m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Container created 53m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Startup probe error: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Startup probe failed: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 53m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv 53m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled up replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 4 to 5 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 5; reason: 53m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Container created 53m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Add eth0 [10.133.0.70/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Container started 53m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Startup probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Startup probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 2 to 3 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 53m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Container created 53m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Add eth0 [10.133.0.71/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Container started 52m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Startup probe failed: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Startup probe error: Get "http://10.133.0.71:15021/healthz/ready": dial tcp 10.133.0.71:15021: connect: connection refused... 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 2 to 3 52m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 52m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Add eth0 [10.132.0.90/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Container created 52m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Container started 52m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Startup probe error: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Startup probe failed: Get "http://10.132.0.90:15021/healthz/ready": dial tcp 10.132.0.90:15021: connect: connection refused 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n 52m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 3 to 6 52m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 3 to 4 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 3 to 4 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q 52m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Container created 52m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Container started 52m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Add eth0 [10.133.0.74/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Add eth0 [10.134.0.71/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Container started 52m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Container created 52m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Container started 52m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Container created 52m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Add eth0 [10.132.0.92/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Container started 52m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Container started 52m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Container created 52m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Add eth0 [10.134.0.73/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Container created 52m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Add eth0 [10.134.0.72/23] from ovn-kubernetes 52m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Startup probe failed: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Startup probe error: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Startup probe error: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Startup probe failed: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused 52m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 7; reason: All metrics below target 52m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 6 to 7 52m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Container started 52m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Container created 52m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Add eth0 [10.134.0.74/23] from ovn-kubernetes 52m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 52m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 3 to 6 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Created pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh 52m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled up replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 2 to 4 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Created pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct 52m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md 52m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 52m Normal Created pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Container created 52m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Container created 52m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Add eth0 [10.133.0.79/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Add eth0 [10.133.0.78/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Add eth0 [10.132.0.95/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Container started 52m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Container started 52m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Container created 52m Normal Created pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Container created 52m Normal Pulled pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Add eth0 [10.134.0.75/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Container created 52m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Container started 52m Normal AddedInterface pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Add eth0 [10.132.0.94/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Container started 52m Normal Started pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Container started 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f (combined from similar events): Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled up replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 5 to 10 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq 52m Normal SuccessfulCreate replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Created pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 52m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Add eth0 [10.133.0.81/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Container created 52m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Container started 52m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Add eth0 [10.132.0.97/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Add eth0 [10.134.0.76/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Container started 52m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Add eth0 [10.132.0.96/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Container started 52m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Container created 52m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Container created 52m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Container started 52m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Container created 52m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Container created 52m Normal AddedInterface pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Add eth0 [10.133.0.82/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Container started 52m Normal Pulled pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Startup probe failed: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Startup probe error: Get "http://10.134.0.75:15021/healthz/ready": dial tcp 10.134.0.75:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Startup probe failed: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Startup probe error: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Startup probe error: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Startup probe failed: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused 52m Warning Unhealthy pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Startup probe failed: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Startup probe error: Get "http://10.132.0.94:15021/healthz/ready": dial tcp 10.132.0.94:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Startup probe failed: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Startup probe error: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Startup probe error: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Startup probe failed: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Startup probe failed: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Startup probe error: Get "http://10.132.0.96:15021/healthz/ready": dial tcp 10.132.0.96:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Startup probe failed: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Startup probe error: Get "http://10.132.0.97:15021/healthz/ready": dial tcp 10.132.0.97:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 7; reason: All metrics below target 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 6 to 7 51m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Container created 51m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Add eth0 [10.134.0.77/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Container started 51m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled up replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 7 to 10 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 4 to 8 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl 51m Normal SuccessfulCreate replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 (combined from similar events): Created pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Container started 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Add eth0 [10.133.0.85/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Add eth0 [10.134.0.79/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled up replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 2 to 3 51m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Container started 51m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Container created 51m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Container created 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Add eth0 [10.134.0.78/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Container created 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Container started 51m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Add eth0 [10.133.0.87/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Container created 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k 51m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Container started 51m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Container created 51m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Add eth0 [10.132.0.99/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Add eth0 [10.133.0.86/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Add eth0 [10.132.0.101/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Container created 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Container started 51m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Container started 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Container started 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Container created 51m Normal Started pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Container started 51m Normal AddedInterface pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Add eth0 [10.132.0.100/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Container created 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Readiness probe failed: Get "http://10.133.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Readiness probe error: Get "http://10.133.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Readiness probe error: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Readiness probe failed: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Readiness probe failed: Get "http://10.132.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Readiness probe failed: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Readiness probe error: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Readiness probe error: Get "http://10.132.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Startup probe failed: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Startup probe failed: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Startup probe error: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Startup probe error: Get "http://10.133.0.87:15021/healthz/ready": dial tcp 10.133.0.87:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Startup probe error: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Startup probe failed: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Startup probe failed: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Startup probe failed: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Startup probe error: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Startup probe failed: Get "http://10.132.0.99:15021/healthz/ready": dial tcp 10.132.0.99:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Startup probe error: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Startup probe error: Get "http://10.133.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Startup probe failed: Get "http://10.133.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 10; reason: All metrics below target 51m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Startup probe error: Get "http://10.132.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd 51m Normal SuccessfulCreate replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 (combined from similar events): Created pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled up replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 8 to 10 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Container created 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Container started 51m Normal Started pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Container started 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Add eth0 [10.134.0.80/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Add eth0 [10.132.0.102/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Container created 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Startup probe failed: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Startup probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Startup probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Startup probe error: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused... 51m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled up replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 7 to 10 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 (combined from similar events): Created pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx 51m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Container created 51m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Add eth0 [10.134.0.81/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Container started 51m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Add eth0 [10.133.0.89/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Add eth0 [10.132.0.103/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Container created 51m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Container started 51m Normal Pulled pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Container created 51m Normal Started pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Container started 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 51m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 4 to 8 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Container started 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Container started 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Container created 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Container started 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Add eth0 [10.133.0.91/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Add eth0 [10.134.0.82/23] from ovn-kubernetes 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Add eth0 [10.132.0.104/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Container created 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Container created 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Container started 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Container created 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Add eth0 [10.133.0.92/23] from ovn-kubernetes 51m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Startup probe failed: Get "http://10.132.0.103:15021/healthz/ready": dial tcp 10.132.0.103:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Startup probe failed: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Startup probe error: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Startup probe failed: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Startup probe error: Get "http://10.132.0.103:15021/healthz/ready": dial tcp 10.132.0.103:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Startup probe error: Get "http://10.133.0.89:15021/healthz/ready": dial tcp 10.133.0.89:15021: connect: connection refused... 51m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 51m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-d76xz 51m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 51m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Add eth0 [10.132.0.105/23] from ovn-kubernetes 51m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Startup probe error: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Startup probe failed: Get "http://10.132.0.104:15021/healthz/ready": dial tcp 10.132.0.104:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Readiness probe failed: Get "http://10.133.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Readiness probe error: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Readiness probe error: Get "http://10.133.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Readiness probe error: Get "http://10.134.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Readiness probe error: Get "http://10.132.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Readiness probe failed: Get "http://10.134.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Readiness probe failed: Get "http://10.132.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Readiness probe failed: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Readiness probe error: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Readiness probe error: Get "http://10.133.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Readiness probe failed: Get "http://10.133.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Startup probe error: Get "http://10.133.0.92:15021/healthz/ready": dial tcp 10.133.0.92:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Readiness probe error: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Readiness probe failed: Get "http://10.132.0.97:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Startup probe failed: Get "http://10.133.0.92:15021/healthz/ready": dial tcp 10.133.0.92:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Readiness probe failed: Get "http://10.132.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Readiness probe error: Get "http://10.132.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Readiness probe failed: Get "http://10.134.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Readiness probe error: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Readiness probe failed: Get "http://10.134.0.74:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Readiness probe error: Get "http://10.134.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Readiness probe failed: Get "http://10.134.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Readiness probe error: Get "http://10.134.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Startup probe failed: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Startup probe error: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Startup probe error: Get "http://10.132.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Startup probe failed: Get "http://10.132.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 7.088s (7.088s including waiting). Image size: 185476477 bytes. 51m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Container created 51m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled up replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 8 to 10 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 (combined from similar events): Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 51m Normal SuccessfulCreate replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Created pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 10; reason: All metrics below target 51m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Container started 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Container started 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Container created 51m Normal Started pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Container started 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Add eth0 [10.132.0.106/23] from ovn-kubernetes 51m Normal Pulled pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Add eth0 [10.134.0.83/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Container created 50m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Startup probe failed: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Startup probe error: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Startup probe failed: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Startup probe error: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused... 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled up replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 3 to 6 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj 50m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg 50m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Add eth0 [10.133.0.98/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Container created 50m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Container started 50m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Add eth0 [10.134.0.85/23] from ovn-kubernetes 50m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Container created 50m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Container created 50m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Container started 50m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Container started 50m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Add eth0 [10.132.0.107/23] from ovn-kubernetes 50m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Startup probe error: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Startup probe failed: Get "http://10.132.0.107:15021/healthz/ready": dial tcp 10.132.0.107:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Startup probe failed: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Startup probe error: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Startup probe failed: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Startup probe error: Get "http://10.134.0.85:15021/healthz/ready": dial tcp 10.134.0.85:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Readiness probe failed: Get "http://10.132.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Readiness probe error: Get "http://10.132.0.104:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Readiness probe failed: Get "http://10.134.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Readiness probe error: Get "http://10.134.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Normal AddedLabel gateway/e2e-ait-cbccff90 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-cbccff90 49m Normal ScalingReplicaSet deployment/e2e-ait-cbccff90-openshift-default Scaled up replica set e2e-ait-cbccff90-openshift-default-69795d4b9 from 0 to 1 49m Normal SuccessfulCreate replicaset/e2e-ait-cbccff90-openshift-default-69795d4b9 Created pod: e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb 49m Normal AddedInterface pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Add eth0 [10.134.0.88/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-cbccff90 Scaled up replica set payload-processing-e2e-ait-cbccff90-5466848f67 from 0 to 1 49m Warning FailedCreate replicaset/payload-processing-e2e-ait-cbccff90-5466848f67 Error creating: pods "payload-processing-e2e-ait-cbccff90-5466848f67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-cbccff90: serviceaccount "payload-processing-e2e-ait-cbccff90" not found 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-cbccff90 Scaled up replica set payload-pre-processing-e2e-ait-cbccff90-5bb79db679 from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-cbccff90-5bb79db679 Created pod: payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-cbccff90-5bb79db679 Error creating: pods "payload-pre-processing-e2e-ait-cbccff90-5bb79db679-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-cbccff90: serviceaccount "payload-processing-e2e-ait-cbccff90" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-cbccff90-5466848f67 Created pod: payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 (combined from similar events): Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv 49m Normal SuccessfulCreate replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Created pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 49m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 49m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled up replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 6 to 10 49m Normal SuccessfulCreate replicaset/e2e-derive-1d321437-openshift-default-5d9954b669 Created pod: e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 49m Normal AddedLabel gateway/e2e-derive-1d321437 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-1d321437 49m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Readiness probe error: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Readiness probe failed: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal ScalingReplicaSet deployment/e2e-derive-1d321437-openshift-default Scaled up replica set e2e-derive-1d321437-openshift-default-5d9954b669 from 0 to 1 49m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Add eth0 [10.133.0.114/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Add eth0 [10.133.0.113/23] from ovn-kubernetes 49m Normal AddedInterface pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Add eth0 [10.134.0.89/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/e2e-ait-42dad67e-openshift-default Scaled up replica set e2e-ait-42dad67e-openshift-default-6c58c8b8c9 from 0 to 1 49m Normal AddedLabel gateway/e2e-ait-42dad67e Added label istio.io/rev=openshift-gateway to gateway e2e-ait-42dad67e 49m Normal SuccessfulCreate replicaset/e2e-ait-42dad67e-openshift-default-6c58c8b8c9 Created pod: e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 49m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Container created 49m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Container started 49m Normal AddedInterface pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Add eth0 [10.133.0.115/23] from ovn-kubernetes 49m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Readiness probe failed: Get "http://10.134.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Container created 49m Normal Pulled pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Container created 49m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Container started 49m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Add eth0 [10.132.0.108/23] from ovn-kubernetes 49m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Readiness probe error: Get "http://10.134.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Readiness probe failed: Get "http://10.134.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Container started 49m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Readiness probe error: Get "http://10.134.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Normal AddedInterface pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Add eth0 [10.134.0.90/23] from ovn-kubernetes 49m Normal Created pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Container created 49m Normal Created pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Container created 49m Normal AddedInterface pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Add eth0 [10.134.0.92/23] from ovn-kubernetes 49m Normal Pulled pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Container created 49m Normal Started pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Container started 49m Normal Pulled pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Add eth0 [10.134.0.91/23] from ovn-kubernetes 49m Normal Started pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Container started 49m Normal Started pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Container started 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-1d321437 Scaled up replica set payload-pre-processing-e2e-derive-1d321437-8d8777567 from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-1d321437 Scaled up replica set payload-processing-e2e-derive-1d321437-6ddc96fcd5 from 0 to 1 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-1d321437-8d8777567 Error creating: pods "payload-pre-processing-e2e-derive-1d321437-8d8777567-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-1d321437: serviceaccount "payload-processing-e2e-derive-1d321437" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-1d321437-6ddc96fcd5 Created pod: payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 49m Warning FailedCreate replicaset/payload-processing-e2e-derive-1d321437-6ddc96fcd5 Error creating: pods "payload-processing-e2e-derive-1d321437-6ddc96fcd5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-1d321437: serviceaccount "payload-processing-e2e-derive-1d321437" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-1d321437-8d8777567 Created pod: payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 49m Normal Created pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Container created 49m Normal Started pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Container started 49m Normal Pulled pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Add eth0 [10.133.0.118/23] from ovn-kubernetes 49m Normal AddedInterface pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Add eth0 [10.134.0.94/23] from ovn-kubernetes 49m Normal Pulled pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Container started 49m Normal Created pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Container created 49m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Startup probe error: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused... 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-42dad67e Scaled up replica set payload-pre-processing-e2e-ait-42dad67e-8559ff9c95 from 0 to 1 49m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Startup probe failed: Get "http://10.133.0.113:15021/healthz/ready": dial tcp 10.133.0.113:15021: connect: connection refused 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95 Created pod: payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb 49m Warning FailedCreate replicaset/payload-processing-e2e-ait-42dad67e-bc59db65c Error creating: pods "payload-processing-e2e-ait-42dad67e-bc59db65c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-42dad67e: serviceaccount "payload-processing-e2e-ait-42dad67e" not found 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95 Error creating: pods "payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-42dad67e: serviceaccount "payload-processing-e2e-ait-42dad67e" not found 49m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Startup probe failed: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-42dad67e Scaled up replica set payload-processing-e2e-ait-42dad67e-bc59db65c from 0 to 1 49m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Startup probe failed: Get "http://10.132.0.108:15021/healthz/ready": dial tcp 10.132.0.108:15021: connect: connection refused 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-42dad67e-bc59db65c Created pod: payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 49m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Startup probe error: Get "http://10.132.0.108:15021/healthz/ready": dial tcp 10.132.0.108:15021: connect: connection refused... 49m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Startup probe error: Get "http://10.133.0.113:15021/healthz/ready": dial tcp 10.133.0.113:15021: connect: connection refused... 49m Normal Created pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Container created 49m Normal Started pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Container started 49m Normal AddedInterface pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Add eth0 [10.133.0.119/23] from ovn-kubernetes 49m Normal AddedInterface pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Add eth0 [10.133.0.120/23] from ovn-kubernetes 49m Normal Started pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Container started 49m Normal Created pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Container created 49m Normal Pulled pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Pulled pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused 49m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused... 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-1d321437-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-1d321437-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-42dad67e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-42dad67e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Add eth0 [10.133.0.121/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-lg2gf 49m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 49m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 49m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-cbccff90-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-cbccff90-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Container created 49m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.446s (2.446s including waiting). Image size: 185476477 bytes. 49m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Container started 49m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Readiness probe error: HTTP probe failed with statuscode: 503... 49m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Readiness probe failed: HTTP probe failed with statuscode: 503 48m Normal Killing pod/payload-pre-processing-e2e-ait-cbccff90-5bb79db679-qx52j Stopping container payload-pre-processing 48m Normal Killing pod/payload-processing-e2e-ait-cbccff90-5466848f67-tx6zf Stopping container payload-processing 48m Normal Pulled pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Container created 48m Normal Started pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Container started 48m Warning ProbeError pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Readiness probe error: Get "http://10.134.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Warning Unhealthy pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Readiness probe failed: Get "http://10.134.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Warning ProbeError pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Readiness probe error: Get "http://10.134.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Warning Unhealthy pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Readiness probe failed: Get "http://10.134.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Normal Pulled pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Killing pod/payload-processing-e2e-derive-1d321437-6ddc96fcd5-h4dg4 Stopping container payload-processing 48m Normal Created pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Container created 48m Normal Started pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Container started 48m Normal Started pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Container started 48m Normal Created pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Container created 48m Normal Pulled pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Killing pod/payload-pre-processing-e2e-derive-1d321437-8d8777567-7t6b4 Stopping container payload-pre-processing 48m Warning Unhealthy pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Startup probe failed: Get "http://10.133.0.115:15021/healthz/ready": dial tcp 10.133.0.115:15021: connect: connection refused 48m Warning ProbeError pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Startup probe error: Get "http://10.133.0.115:15021/healthz/ready": dial tcp 10.133.0.115:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Startup probe failed: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-1d321437-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-1d321437-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal Killing pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Stopping container istio-proxy 48m Warning ProbeError pod/e2e-ait-cbccff90-openshift-default-69795d4b9-wp4jb Startup probe error: Get "http://10.134.0.88:15021/healthz/ready": dial tcp 10.134.0.88:15021: connect: connection refused... 48m Normal Killing pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Stopping container payload-pre-processing 48m Normal Killing pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Stopping container payload-processing 48m Normal AddedLabel gateway/e2e-migrate-1bd84be0-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-1bd84be0-gw 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-42dad67e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-42dad67e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal SuccessfulCreate replicaset/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85 Created pod: e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 48m Normal ScalingReplicaSet deployment/e2e-migrate-1bd84be0-gw-openshift-default Scaled up replica set e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85 from 0 to 1 48m Normal AddedInterface pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Add eth0 [10.134.0.98/23] from ovn-kubernetes 48m Normal Pulled pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Container created 48m Normal Started pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Container started 48m Warning ProbeError pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Startup probe error: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Startup probe failed: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused 48m Normal Killing pod/e2e-derive-1d321437-openshift-default-5d9954b669-rqw94 Stopping container istio-proxy 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-1bd84be0-5b467646b5 Created pod: payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp 48m Warning Unhealthy pod/payload-processing-e2e-ait-42dad67e-bc59db65c-s5wk4 Readiness probe failed: timeout: failed to connect service "10.133.0.120:50052" within 1s: context deadline exceeded 48m Warning FailedCreate replicaset/payload-processing-e2e-migrate-1bd84be0-5b467646b5 Error creating: pods "payload-processing-e2e-migrate-1bd84be0-5b467646b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-1bd84be0: serviceaccount "payload-processing-e2e-migrate-1bd84be0" not found 48m Warning Unhealthy pod/payload-pre-processing-e2e-ait-42dad67e-8559ff9c95-dsjwb Readiness probe failed: timeout: failed to connect service "10.133.0.119:50052" within 1s: context deadline exceeded 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-1bd84be0 Scaled up replica set payload-processing-e2e-migrate-1bd84be0-5b467646b5 from 0 to 1 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f Error creating: pods "payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-1bd84be0: serviceaccount "payload-processing-e2e-migrate-1bd84be0" not found 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f Created pod: payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-1bd84be0 Scaled up replica set payload-pre-processing-e2e-migrate-1bd84be0-59d485677f from 0 to 1 48m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Add eth0 [10.134.0.99/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Container created 48m Normal AddedInterface pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Add eth0 [10.134.0.100/23] from ovn-kubernetes 48m Normal Pulled pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Container started 48m Normal Started pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Container started 48m Normal Created pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Container created 48m Normal Killing pod/e2e-ait-42dad67e-openshift-default-6c58c8b8c9-zm922 Stopping container istio-proxy 48m Normal AddedInterface pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Add eth0 [10.134.0.101/23] from ovn-kubernetes 48m Normal AddedLabel gateway/e2e-ait-5d6f1188 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-5d6f1188 48m Normal ScalingReplicaSet deployment/e2e-ait-5d6f1188-openshift-default Scaled up replica set e2e-ait-5d6f1188-openshift-default-fb4bfb8b7 from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7 Created pod: e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-5d6f1188 Scaled up replica set payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7 from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7 Created pod: payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-5d6f1188 Scaled up replica set payload-processing-e2e-ait-5d6f1188-75bb67cdcc from 0 to 1 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7 Error creating: pods "payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-5d6f1188: serviceaccount "payload-processing-e2e-ait-5d6f1188" not found 48m Warning FailedCreate replicaset/payload-processing-e2e-ait-5d6f1188-75bb67cdcc Error creating: pods "payload-processing-e2e-ait-5d6f1188-75bb67cdcc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-5d6f1188: serviceaccount "payload-processing-e2e-ait-5d6f1188" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-5d6f1188-75bb67cdcc Created pod: payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s 48m Normal AddedInterface pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Add eth0 [10.134.0.102/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Container created 48m Normal AddedInterface pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Add eth0 [10.134.0.103/23] from ovn-kubernetes 48m Normal Started pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Container started 48m Normal Started pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Container started 48m Normal Created pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Container created 48m Normal Pulled pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-5d6f1188-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-5d6f1188-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-1bd84be0-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-1bd84be0-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Killing pod/payload-pre-processing-e2e-migrate-1bd84be0-59d485677f-22m7h Stopping container payload-pre-processing 47m Normal Killing pod/payload-processing-e2e-migrate-1bd84be0-5b467646b5-hlzdp Stopping container payload-processing 47m Normal Killing pod/e2e-migrate-1bd84be0-gw-openshift-default-7f8dfd6d85-pnrh8 Stopping container istio-proxy 47m Warning ProbeError pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Readiness probe error: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 47m Warning Unhealthy pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Readiness probe failed: Get "http://10.134.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 47m Normal Created pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Container created 47m Normal Pulled pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Started pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Container started 47m Warning Unhealthy pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Startup probe failed: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused 47m Warning ProbeError pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Startup probe error: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused... 47m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-25ff27-openshift-default Scaled up replica set e2e-shared-a-w3-25ff27-openshift-default-856fc89989 from 0 to 1 47m Normal AddedLabel gateway/e2e-shared-a-w3-25ff27 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-25ff27 47m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-25ff27-openshift-default-856fc89989 Created pod: e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr 47m Normal AddedInterface pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Add eth0 [10.134.0.106/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/e2e-mt-987d6f93-openshift-default Scaled up replica set e2e-mt-987d6f93-openshift-default-58f9fd5f6f from 0 to 1 47m Normal AddedLabel gateway/e2e-mt-987d6f93 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-987d6f93 47m Normal SuccessfulCreate replicaset/e2e-mt-987d6f93-openshift-default-58f9fd5f6f Created pod: e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl 47m Normal AddedInterface pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Add eth0 [10.134.0.107/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-25ff27 Scaled up replica set payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99 Created pod: payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-25ff27 Scaled up replica set payload-processing-e2e-shared-a-w3-25ff27-778dfd8567 from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567 Error creating: pods "payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-25ff27: serviceaccount "payload-processing-e2e-shared-a-w3-25ff27" not found 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-25ff27: serviceaccount "payload-processing-e2e-shared-a-w3-25ff27" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567 Created pod: payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s 47m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Container created 47m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Add eth0 [10.134.0.109/23] from ovn-kubernetes 47m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Container started 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Add eth0 [10.134.0.108/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Container created 47m Normal Started pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Container started 47m Normal Killing pod/payload-processing-e2e-ait-5d6f1188-75bb67cdcc-lzz5s Stopping container payload-processing 47m Normal Killing pod/payload-pre-processing-e2e-ait-5d6f1188-8678b9f9d7-8p45g Stopping container payload-pre-processing 47m Normal SuccessfulDelete replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Deleted pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default New size: 3; reason: All metrics below target 47m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled down replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 4 to 3 47m Normal Killing pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-5rjct Stopping container istio-proxy 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-987d6f93-c846f88c5 Created pod: payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-987d6f93 Scaled up replica set payload-processing-e2e-mt-987d6f93-c846f88c5 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-987d6f93-86f47b655 Created pod: payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-5d6f1188-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-987d6f93-86f47b655 Error creating: pods "payload-pre-processing-e2e-mt-987d6f93-86f47b655-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-987d6f93: serviceaccount "payload-processing-e2e-mt-987d6f93" not found 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-5d6f1188-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedCreate replicaset/payload-processing-e2e-mt-987d6f93-c846f88c5 Error creating: pods "payload-processing-e2e-mt-987d6f93-c846f88c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-987d6f93: serviceaccount "payload-processing-e2e-mt-987d6f93" not found 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-987d6f93 Scaled up replica set payload-pre-processing-e2e-mt-987d6f93-86f47b655 from 0 to 1 47m Normal Created pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Container created 47m Normal AddedInterface pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Add eth0 [10.134.0.111/23] from ovn-kubernetes 47m Normal Started pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Container started 47m Normal Started pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Container started 47m Normal Created pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Container created 47m Normal Pulled pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Add eth0 [10.134.0.110/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Killing pod/e2e-ait-5d6f1188-openshift-default-fb4bfb8b7-gxd67 Stopping container istio-proxy 46m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c Created pod: e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs 46m Normal AddedLabel gateway/e2e-shared-a-w6-83a98b Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-83a98b 46m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-83a98b-openshift-default Scaled up replica set e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c from 0 to 1 46m Normal AddedInterface pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Add eth0 [10.134.0.113/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9 Created pod: payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9 Error creating: pods "payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-83a98b: serviceaccount "payload-processing-e2e-shared-a-w6-83a98b" not found 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-83a98b Scaled up replica set payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9 from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-83a98b Scaled up replica set payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f Created pod: payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f Error creating: pods "payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-83a98b: serviceaccount "payload-processing-e2e-shared-a-w6-83a98b" not found 46m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Add eth0 [10.133.0.129/23] from ovn-kubernetes 46m Normal Pulled pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Container created 46m Normal Started pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Container started 46m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Container created 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Add eth0 [10.134.0.114/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Container started 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default New size: 2; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Deleted pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh 46m Normal Killing pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-wstgh Stopping container istio-proxy 46m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled down replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 3 to 2 46m Normal AddedLabel gateway/e2e-shared-b-w3-ba4687 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-ba4687 46m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-ba4687-openshift-default Scaled up replica set e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c Created pod: e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb 46m Normal AddedInterface pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Add eth0 [10.134.0.115/23] from ovn-kubernetes 46m Normal Created pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Container created 46m Normal Pulled pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Container started 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-ba4687 Scaled up replica set payload-processing-e2e-shared-b-w3-ba4687-7957fb899c from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-ba4687 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c Error creating: pods "payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-ba4687: serviceaccount "payload-processing-e2e-shared-b-w3-ba4687" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf Error creating: pods "payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-ba4687: serviceaccount "payload-processing-e2e-shared-b-w3-ba4687" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf Created pod: payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c Created pod: payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz 46m Normal Pulled pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Add eth0 [10.133.0.131/23] from ovn-kubernetes 46m Normal Started pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Container started 46m Normal Created pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Container created 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Add eth0 [10.134.0.116/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Container started 46m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Container created 46m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Warning Unhealthy pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Startup probe failed: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused 46m Warning ProbeError pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Startup probe error: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused... 46m Normal AddedLabel gateway/e2e-shared-a-w1-e2dfd6 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-e2dfd6 46m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9 Created pod: e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz 46m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-e2dfd6-openshift-default Scaled up replica set e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9 from 0 to 1 46m Normal AddedInterface pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Add eth0 [10.133.0.132/23] from ovn-kubernetes 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-25ff27-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-25ff27-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-ba4687-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-ba4687-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal Started pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Container started 46m Normal Created pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Container created 46m Normal Pulled pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f Error creating: pods "payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-e2dfd6: serviceaccount "payload-processing-e2e-shared-a-w1-e2dfd6" not found 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-e2dfd6 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f Created pod: payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf Error creating: pods "payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-e2dfd6: serviceaccount "payload-processing-e2e-shared-a-w1-e2dfd6" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf Created pod: payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-e2dfd6 Scaled up replica set payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f from 0 to 1 45m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Add eth0 [10.133.0.133/23] from ovn-kubernetes 45m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Pulled pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Container started 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Add eth0 [10.134.0.117/23] from ovn-kubernetes 45m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Container created 45m Normal Created pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Container created 45m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Container started 45m Warning ProbeError pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Startup probe error: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Startup probe failed: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused 45m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-c839e5-openshift-default Scaled up replica set e2e-shared-a-w0-c839e5-openshift-default-6c446cf677 from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677 Created pod: e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx 45m Normal AddedLabel gateway/e2e-shared-a-w0-c839e5 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-c839e5 45m Normal AddedInterface pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Add eth0 [10.134.0.118/23] from ovn-kubernetes 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-987d6f93-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-987d6f93-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm Stopping container istio-proxy 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 8; reason: All metrics below target 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled down replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 10 to 8 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-64mwm 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf 45m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8jmrf Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-c839e5 Scaled up replica set payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9 from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-c839e5 Scaled up replica set payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99 from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9 Error creating: pods "payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-c839e5: serviceaccount "payload-processing-e2e-shared-a-w0-c839e5" not found 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99 Created pod: payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9 Created pod: payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f 45m Normal AddedLabel gateway/e2e-shared-b-w6-1d454b Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-1d454b 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99 Error creating: pods "payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-c839e5: serviceaccount "payload-processing-e2e-shared-a-w0-c839e5" not found 45m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Container created 45m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Container started 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Add eth0 [10.134.0.119/23] from ovn-kubernetes 45m Normal AddedInterface pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Add eth0 [10.133.0.135/23] from ovn-kubernetes 45m Normal Created pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Container created 45m Normal Pulled pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-1d454b-openshift-default Scaled up replica set e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5 from 0 to 1 45m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Container started 45m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Add eth0 [10.133.0.134/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5 Created pod: e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g 45m Normal ScalingReplicaSet deployment/e2e-mt-47a126f3-openshift-default Scaled up replica set e2e-mt-47a126f3-openshift-default-5789f86f77 from 0 to 1 45m Normal AddedLabel gateway/e2e-mt-47a126f3 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-47a126f3 45m Normal SuccessfulCreate replicaset/e2e-mt-47a126f3-openshift-default-5789f86f77 Created pod: e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s 45m Normal AddedInterface pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Add eth0 [10.134.0.121/23] from ovn-kubernetes 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-1d454b Scaled up replica set payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7 from 0 to 1 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7 Error creating: pods "payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-1d454b: serviceaccount "payload-processing-e2e-shared-b-w6-1d454b" not found 45m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7 Error creating: pods "payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-1d454b: serviceaccount "payload-processing-e2e-shared-b-w6-1d454b" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7 Created pod: payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-1d454b Scaled up replica set payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7 from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7 Created pod: payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q 45m Normal Pulled pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Add eth0 [10.134.0.123/23] from ovn-kubernetes 45m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Add eth0 [10.134.0.122/23] from ovn-kubernetes 45m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Container created 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-987d6f93-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal Created pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Container created 45m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Container started 45m Normal Started pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Container started 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-987d6f93-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning Unhealthy pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Readiness probe failed: Get "http://10.134.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-kl266 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 6; reason: All metrics below target 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled down replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 8 to 6 45m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-4tmvf Stopping container istio-proxy 45m Warning ProbeError pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Readiness probe error: Get "http://10.134.0.115:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Warning ProbeError pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Readiness probe error: Get "http://10.134.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Warning Unhealthy pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Readiness probe failed: Get "http://10.134.0.113:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Normal Created pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Container created 45m Normal Pulled pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Container started 45m Normal Pulled pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Container created 45m Normal Started pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Container started 45m Warning ProbeError pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Startup probe error: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Startup probe failed: Get "http://10.134.0.115:15021/healthz/ready": dial tcp 10.134.0.115:15021: connect: connection refused 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67 Error creating: pods "payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-47a126f3: serviceaccount "payload-processing-e2e-mt-47a126f3" not found 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-47a126f3 Scaled up replica set payload-pre-processing-e2e-mt-47a126f3-5b7b569d67 from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-mt-47a126f3-6f66b665c9 Error creating: pods "payload-processing-e2e-mt-47a126f3-6f66b665c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-47a126f3: serviceaccount "payload-processing-e2e-mt-47a126f3" not found 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-47a126f3 Scaled up replica set payload-processing-e2e-mt-47a126f3-6f66b665c9 from 0 to 1 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-47a126f3-6f66b665c9 Created pod: payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw 45m Normal AddedInterface pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Add eth0 [10.133.0.136/23] from ovn-kubernetes 45m Normal Pulled pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Warning Unhealthy pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Startup probe failed: Get "http://10.134.0.113:15021/healthz/ready": dial tcp 10.134.0.113:15021: connect: connection refused 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67 Created pod: payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml 45m Normal Created pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Container created 45m Normal Started pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Container started 45m Warning ProbeError pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Startup probe error: Get "http://10.134.0.113:15021/healthz/ready": dial tcp 10.134.0.113:15021: connect: connection refused... 45m Normal Pulled pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Container started 45m Normal Created pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Container created 45m Normal AddedInterface pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Add eth0 [10.134.0.124/23] from ovn-kubernetes 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-83a98b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-83a98b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49 Created pod: e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw 45m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-a5faaa-openshift-default Scaled up replica set e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49 from 0 to 1 45m Normal AddedLabel gateway/e2e-shared-b-w1-a5faaa Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-a5faaa 45m Normal AddedInterface pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Add eth0 [10.133.0.137/23] from ovn-kubernetes 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-83a98b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-83a98b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Add eth0 [10.133.0.138/23] from ovn-kubernetes 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5 Error creating: pods "payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-a5faaa: serviceaccount "payload-processing-e2e-shared-b-w1-a5faaa" not found 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-a5faaa Scaled up replica set payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc Created pod: payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc Error creating: pods "payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-a5faaa: serviceaccount "payload-processing-e2e-shared-b-w1-a5faaa" not found 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5 Created pod: payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-a5faaa Scaled up replica set payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5 from 0 to 1 44m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Container created 44m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Add eth0 [10.133.0.139/23] from ovn-kubernetes 44m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Container started 44m Normal Pulled pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Container created 44m Normal Started pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Container started 44m Normal Started pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Container started 44m Normal Pulled pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Container created 44m Warning Unhealthy pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Startup probe failed: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused 44m Warning ProbeError pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Startup probe error: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused... 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-ba4687-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-e2dfd6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-e2dfd6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-ba4687-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 6; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq 44m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk 44m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled down replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 10 to 6 44m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-bbqhq Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-cfjnk Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-x96dv Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-9zmmg Stopping container istio-proxy 44m Normal Killing pod/payload-pre-processing-e2e-mt-987d6f93-86f47b655-m6l5k Stopping container payload-pre-processing 44m Normal Killing pod/payload-processing-e2e-mt-987d6f93-c846f88c5-7b7hn Stopping container payload-processing 44m Warning Unhealthy pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Readiness probe failed: Get "http://10.133.0.135:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Warning ProbeError pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Readiness probe error: Get "http://10.133.0.135:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Normal Pulled pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Container created 44m Normal Started pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Container started 44m Normal ScalingReplicaSet deployment/e2e-api-a-a37c29-openshift-default Scaled up replica set e2e-api-a-a37c29-openshift-default-77fdcb54f5 from 0 to 1 44m Normal AddedLabel gateway/e2e-api-a-a37c29 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-a37c29 44m Normal SuccessfulCreate replicaset/e2e-api-a-a37c29-openshift-default-77fdcb54f5 Created pod: e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k 44m Normal Pulled pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Container created 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-c839e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-c839e5-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal Started pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Container started 44m Normal AddedInterface pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Add eth0 [10.133.0.141/23] from ovn-kubernetes 44m Warning Unhealthy pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Startup probe failed: Get "http://10.133.0.135:15021/healthz/ready": dial tcp 10.133.0.135:15021: connect: connection refused 44m Warning ProbeError pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Startup probe error: Get "http://10.133.0.135:15021/healthz/ready": dial tcp 10.133.0.135:15021: connect: connection refused... 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-a37c29 Scaled up replica set payload-pre-processing-e2e-api-a-a37c29-798fb4cd67 from 0 to 1 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-a37c29 Scaled up replica set payload-processing-e2e-api-a-a37c29-58597475bf from 0 to 1 44m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7 Created pod: e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-a37c29-58597475bf Created pod: payload-processing-e2e-api-a-a37c29-58597475bf-sg855 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67 Error creating: pods "payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a37c29: serviceaccount "payload-processing-e2e-api-a-a37c29" not found 44m Normal AddedLabel gateway/e2e-shared-b-w0-e69af1 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-e69af1 44m Warning FailedCreate replicaset/payload-processing-e2e-api-a-a37c29-58597475bf Error creating: pods "payload-processing-e2e-api-a-a37c29-58597475bf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a37c29: serviceaccount "payload-processing-e2e-api-a-a37c29" not found 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67 Created pod: payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh 44m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-e69af1-openshift-default Scaled up replica set e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7 from 0 to 1 44m Normal Pulled pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Add eth0 [10.133.0.142/23] from ovn-kubernetes 44m Warning ProbeError pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Startup probe error: Get "http://10.134.0.118:15021/healthz/ready": dial tcp 10.134.0.118:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Startup probe failed: Get "http://10.134.0.118:15021/healthz/ready": dial tcp 10.134.0.118:15021: connect: connection refused 44m Normal AddedInterface pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Add eth0 [10.133.0.143/23] from ovn-kubernetes 44m Normal Created pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Container created 44m Normal Started pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Container started 44m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Add eth0 [10.134.0.125/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Container created 44m Normal Started pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Container started 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-e2dfd6-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-e2dfd6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning ProbeError pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Readiness probe error: Get "http://10.134.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 6; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 44m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled down replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 10 to 6 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt 44m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q 44m Warning Unhealthy pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Readiness probe failed: Get "http://10.134.0.121:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 44m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5nbg6 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f 44m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-pdqvh Stopping container istio-proxy 44m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Readiness probe error: HTTP probe failed with statuscode: 503... 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv 44m Normal Started pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Container started 44m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled down replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 10 to 5 44m Normal Created pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Container created 44m Normal Pulled pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-wj62q Stopping container istio-proxy 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 5; reason: All metrics below target 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl 44m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-x6xd7 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl 44m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-t44zt Readiness probe failed: HTTP probe failed with statuscode: 503 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 6; reason: All metrics below target 44m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled down replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 10 to 6 44m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-9mxnv Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-4j28f Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-hj6cj 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w 44m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-7dtl6 44m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-zlqpl Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-j299z Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-85gpl Stopping container istio-proxy 44m Warning Unhealthy pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning ProbeError pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-mnp2w Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-e69af1-755b48899f Error creating: pods "payload-processing-e2e-shared-b-w0-e69af1-755b48899f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-e69af1: serviceaccount "payload-processing-e2e-shared-b-w0-e69af1" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-e69af1 Scaled up replica set payload-processing-e2e-shared-b-w0-e69af1-755b48899f from 0 to 1 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-e69af1 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc from 0 to 1 44m Normal Started pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Container started 44m Normal Created pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Container created 44m Normal Pulled pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-e69af1-755b48899f Created pod: payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp 44m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Add eth0 [10.133.0.144/23] from ovn-kubernetes 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc Error creating: pods "payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-e69af1: serviceaccount "payload-processing-e2e-shared-b-w0-e69af1" not found 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc Created pod: payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v 44m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Container created 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-1d454b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Add eth0 [10.134.0.126/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Container started 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-1d454b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning Unhealthy pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Startup probe failed: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused 44m Warning ProbeError pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Startup probe error: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused... 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-c839e5-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-c839e5-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal Killing pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Stopping container istio-proxy 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-1d454b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-1d454b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal ScalingReplicaSet deployment/e2e-api-a-c68724-openshift-default Scaled up replica set e2e-api-a-c68724-openshift-default-6b8f8769bf from 0 to 1 43m Normal AddedLabel gateway/e2e-api-a-c68724 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-c68724 43m Normal AddedInterface pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Add eth0 [10.134.0.128/23] from ovn-kubernetes 43m Normal SuccessfulCreate replicaset/e2e-api-a-c68724-openshift-default-6b8f8769bf Created pod: e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm 43m Warning ProbeError pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Readiness probe error: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-mt-987d6f93-openshift-default-58f9fd5f6f-pv6zl Readiness probe failed: Get "http://10.134.0.107:15021/healthz/ready": dial tcp 10.134.0.107:15021: connect: connection refused 43m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 43m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x 43m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7mw5q 43m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-l9j58 Stopping container istio-proxy 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 6; reason: All metrics below target 43m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-2nkg6 43m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-49v5x Stopping container istio-proxy 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-47a126f3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal Started pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Container started 43m Normal Created pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Container created 43m Normal Pulled pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled down replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 10 to 6 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-47a126f3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-a5faaa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-a5faaa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning ProbeError pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Startup probe error: Get "http://10.133.0.137:15021/healthz/ready": dial tcp 10.133.0.137:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Startup probe failed: Get "http://10.133.0.137:15021/healthz/ready": dial tcp 10.133.0.137:15021: connect: connection refused 43m Normal Pulled pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Started pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Container started 43m Normal Created pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Container created 43m Normal AddedLabel gateway/e2e-api-b-250992 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-250992 43m Normal SuccessfulCreate replicaset/e2e-api-b-250992-openshift-default-85d78c677c Created pod: e2e-api-b-250992-openshift-default-85d78c677c-h4md9 43m Normal ScalingReplicaSet deployment/e2e-api-b-250992-openshift-default Scaled up replica set e2e-api-b-250992-openshift-default-85d78c677c from 0 to 1 43m Normal AddedInterface pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Add eth0 [10.134.0.134/23] from ovn-kubernetes 43m Warning Unhealthy pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Startup probe failed: Get "http://10.133.0.141:15021/healthz/ready": dial tcp 10.133.0.141:15021: connect: connection refused 43m Warning ProbeError pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Startup probe error: Get "http://10.133.0.141:15021/healthz/ready": dial tcp 10.133.0.141:15021: connect: connection refused... 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-c68724 Scaled up replica set payload-processing-e2e-api-a-c68724-697d45f96c from 0 to 1 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-a5faaa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-c68724 Scaled up replica set payload-pre-processing-e2e-api-a-c68724-dbd5b895 from 0 to 1 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-a5faaa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedCreate replicaset/payload-processing-e2e-api-a-c68724-697d45f96c Error creating: pods "payload-processing-e2e-api-a-c68724-697d45f96c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-c68724: serviceaccount "payload-processing-e2e-api-a-c68724" not found 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-c68724-697d45f96c Created pod: payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-c68724-dbd5b895 Error creating: pods "payload-pre-processing-e2e-api-a-c68724-dbd5b895-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-c68724: serviceaccount "payload-processing-e2e-api-a-c68724" not found 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-c68724-dbd5b895 Created pod: payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-250992-6d475684bc Error creating: pods "payload-pre-processing-e2e-api-b-250992-6d475684bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-250992: serviceaccount "payload-processing-e2e-api-b-250992" not found 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-250992-6d475684bc Created pod: payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-250992 Scaled up replica set payload-pre-processing-e2e-api-b-250992-6d475684bc from 0 to 1 43m Normal Started pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Container started 43m Normal Created pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Container created 43m Normal Pulled pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Add eth0 [10.134.0.135/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Container started 43m Normal Created pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Container created 43m Normal Pulled pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Add eth0 [10.134.0.136/23] from ovn-kubernetes 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-250992 Scaled up replica set payload-processing-e2e-api-b-250992-5d8ffd9995 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-250992-5d8ffd9995 Created pod: payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb 43m Warning FailedCreate replicaset/payload-processing-e2e-api-b-250992-5d8ffd9995 Error creating: pods "payload-processing-e2e-api-b-250992-5d8ffd9995-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-250992: serviceaccount "payload-processing-e2e-api-b-250992" not found 43m Normal Pulled pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Created pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Container created 43m Normal AddedLabel gateway/e2e-inf-a-5a2c68 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-5a2c68 43m Normal ScalingReplicaSet deployment/e2e-inf-a-5a2c68-openshift-default Scaled up replica set e2e-inf-a-5a2c68-openshift-default-5db97fc557 from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-inf-a-5a2c68-openshift-default-5db97fc557 Created pod: e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j 43m Normal Started pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Container started 43m Normal Created pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Container created 43m Normal AddedInterface pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Add eth0 [10.134.0.137/23] from ovn-kubernetes 43m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Add eth0 [10.134.0.138/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Container started 43m Normal Pulled pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Add eth0 [10.134.0.139/23] from ovn-kubernetes 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-5a2c68 Scaled up replica set payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c from 0 to 1 43m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-5a2c68-59b9485d87 Error creating: pods "payload-processing-e2e-inf-a-5a2c68-59b9485d87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-5a2c68: serviceaccount "payload-processing-e2e-inf-a-5a2c68" not found 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c Error creating: pods "payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-5a2c68: serviceaccount "payload-processing-e2e-inf-a-5a2c68" not found 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-5a2c68-59b9485d87 Created pod: payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-5a2c68 Scaled up replica set payload-processing-e2e-inf-a-5a2c68-59b9485d87 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c Created pod: payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq 43m Normal Created pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Container created 43m Normal AddedInterface pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Add eth0 [10.134.0.140/23] from ovn-kubernetes 43m Normal Pulled pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Created pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Container created 43m Normal Started pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Container started 43m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Add eth0 [10.134.0.141/23] from ovn-kubernetes 43m Normal Pulled pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Started pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Container started 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 5; reason: All metrics below target 43m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx 43m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled down replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 6 to 5 43m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-wmzfx Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled down replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 6 to 5 43m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 5; reason: All metrics below target 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default Scaled down replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 6 to 5 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 5; reason: All metrics below target 43m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr Stopping container istio-proxy 43m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj Stopping container istio-proxy 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qlcmj 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 5; reason: All metrics below target 43m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled down replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 6 to 5 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-f8bnr 43m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-7wzxs Stopping container istio-proxy 42m Normal SuccessfulCreate replicaset/e2e-inf-a-b27079-openshift-default-5f79bb8845 Created pod: e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 42m Normal ScalingReplicaSet deployment/e2e-inf-a-b27079-openshift-default Scaled up replica set e2e-inf-a-b27079-openshift-default-5f79bb8845 from 0 to 1 42m Normal AddedLabel gateway/e2e-inf-a-b27079 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-b27079 42m Normal AddedInterface pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Add eth0 [10.134.0.142/23] from ovn-kubernetes 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-a37c29-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-a37c29-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal Pulled pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Container created 42m Normal Started pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Container started 42m Warning ProbeError pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Startup probe error: Get "http://10.133.0.143:15021/healthz/ready": dial tcp 10.133.0.143:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Startup probe failed: Get "http://10.133.0.143:15021/healthz/ready": dial tcp 10.133.0.143:15021: connect: connection refused 42m Normal SuccessfulCreate replicaset/e2e-mt-47a126f3-openshift-default-5789f86f77 Created pod: e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j 42m Normal ScalingReplicaSet deployment/e2e-mt-47a126f3-openshift-default Scaled up replica set e2e-mt-47a126f3-openshift-default-5789f86f77 from 1 to 2 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-47a126f3-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 42m Normal Started pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Container started 42m Normal Pulled pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Container created 42m Normal AddedInterface pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Add eth0 [10.133.0.147/23] from ovn-kubernetes 42m Warning Unhealthy pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Startup probe failed: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused 42m Warning ProbeError pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Startup probe error: Get "http://10.133.0.147:15021/healthz/ready": dial tcp 10.133.0.147:15021: connect: connection refused... 42m Normal AddedLabel gateway/e2e-api-a-1a771e Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-1a771e 42m Normal SuccessfulCreate replicaset/e2e-api-a-1a771e-openshift-default-d4679577c Created pod: e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk 42m Normal AddedInterface pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Add eth0 [10.134.0.146/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/e2e-api-a-1a771e-openshift-default Scaled up replica set e2e-api-a-1a771e-openshift-default-d4679577c from 0 to 1 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-e69af1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-e69af1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-b27079-64599b455 Error creating: pods "payload-pre-processing-e2e-inf-a-b27079-64599b455-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-b27079: serviceaccount "payload-processing-e2e-inf-a-b27079" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-b27079 Scaled up replica set payload-processing-e2e-inf-a-b27079-bd68fb84c from 0 to 1 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-b27079-bd68fb84c Error creating: pods "payload-processing-e2e-inf-a-b27079-bd68fb84c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-b27079: serviceaccount "payload-processing-e2e-inf-a-b27079" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-b27079-bd68fb84c Created pod: payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-b27079-64599b455 Created pod: payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-b27079 Scaled up replica set payload-pre-processing-e2e-inf-a-b27079-64599b455 from 0 to 1 42m Normal Started pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Container started 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Add eth0 [10.133.0.148/23] from ovn-kubernetes 42m Normal Started pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Container started 42m Normal Pulled pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Container created 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Add eth0 [10.133.0.149/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Container created 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-b27079-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-1a771e-65d78995c5 Created pod: payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-1a771e-58879567c5 Created pod: payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-1a771e-58879567c5 Error creating: pods "payload-pre-processing-e2e-api-a-1a771e-58879567c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-1a771e: serviceaccount "payload-processing-e2e-api-a-1a771e" not found 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-1a771e Scaled up replica set payload-pre-processing-e2e-api-a-1a771e-58879567c5 from 0 to 1 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-1a771e Scaled up replica set payload-processing-e2e-api-a-1a771e-65d78995c5 from 0 to 1 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-b27079-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedCreate replicaset/payload-processing-e2e-api-a-1a771e-65d78995c5 Error creating: pods "payload-processing-e2e-api-a-1a771e-65d78995c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-1a771e: serviceaccount "payload-processing-e2e-api-a-1a771e" not found 42m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Add eth0 [10.133.0.150/23] from ovn-kubernetes 42m Normal Started pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Container started 42m Normal Pulled pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Pulled pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Add eth0 [10.134.0.147/23] from ovn-kubernetes 42m Normal Created pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Container created 42m Normal Created pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Container created 42m Normal Started pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Container started 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-e69af1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-e69af1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Normal ScalingReplicaSet deployment/e2e-api-a-c68724-openshift-default Scaled up replica set e2e-api-a-c68724-openshift-default-6b8f8769bf from 1 to 4 42m Normal SuccessfulCreate replicaset/e2e-api-a-c68724-openshift-default-6b8f8769bf Created pod: e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr 42m Normal AddedInterface pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Add eth0 [10.133.0.152/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/e2e-api-a-c68724-openshift-default-6b8f8769bf Created pod: e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-api-a-c68724-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 42m Normal SuccessfulCreate replicaset/e2e-api-a-c68724-openshift-default-6b8f8769bf Created pod: e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 42m Normal Created pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Container created 42m Normal Started pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Container started 42m Normal Started pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Container started 42m Normal Pulled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Add eth0 [10.132.0.116/23] from ovn-kubernetes 42m Normal Started pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Container started 42m Normal AddedInterface pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Add eth0 [10.134.0.148/23] from ovn-kubernetes 42m Normal Pulled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Container created 42m Normal Created pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Container created 42m Normal Pulled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Container started 42m Normal Created pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Container created 42m Normal Pulled pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Started pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Container started 42m Normal Created pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Container created 42m Normal Pulled pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal AddedLabel gateway/e2e-api-b-e1d6ff Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-e1d6ff 42m Normal ScalingReplicaSet deployment/e2e-api-b-e1d6ff-openshift-default Scaled up replica set e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9 Created pod: e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz 42m Normal AddedInterface pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Add eth0 [10.133.0.153/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Startup probe error: Get "http://10.132.0.116:15021/healthz/ready": dial tcp 10.132.0.116:15021: connect: connection refused... 42m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Startup probe error: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Startup probe failed: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused 42m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Startup probe failed: Get "http://10.132.0.116:15021/healthz/ready": dial tcp 10.132.0.116:15021: connect: connection refused 41m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Startup probe failed: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused 41m Warning ProbeError pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Startup probe error: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused... 41m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Startup probe error: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Startup probe failed: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-e1d6ff Scaled up replica set payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779 from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779 Error creating: pods "payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-e1d6ff: serviceaccount "payload-processing-e2e-api-b-e1d6ff" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-e1d6ff Scaled up replica set payload-processing-e2e-api-b-e1d6ff-b5b9bfd85 from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85 Created pod: payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779 Created pod: payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg 41m Warning FailedCreate replicaset/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85 Error creating: pods "payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-e1d6ff: serviceaccount "payload-processing-e2e-api-b-e1d6ff" not found 41m Normal Pulled pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Add eth0 [10.134.0.149/23] from ovn-kubernetes 41m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Startup probe error: Get "http://10.134.0.128:15021/healthz/ready": dial tcp 10.134.0.128:15021: connect: connection refused... 41m Normal Started pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Container started 41m Normal Created pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Container created 41m Normal Started pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Container started 41m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Startup probe failed: Get "http://10.134.0.128:15021/healthz/ready": dial tcp 10.134.0.128:15021: connect: connection refused 41m Normal Created pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Container created 41m Normal Pulled pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Add eth0 [10.133.0.155/23] from ovn-kubernetes 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-c68724-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-c68724-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal Killing pod/payload-processing-e2e-api-a-a37c29-58597475bf-sg855 Stopping container payload-processing 41m Normal Killing pod/payload-pre-processing-e2e-api-a-a37c29-798fb4cd67-h9rnh Stopping container payload-pre-processing 41m Warning Unhealthy pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Readiness probe failed: Get "http://10.134.0.134:15021/healthz/ready": read tcp 10.134.0.2:49056->10.134.0.134:15021: read: connection reset by peer 41m Warning ProbeError pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Readiness probe error: Get "http://10.134.0.134:15021/healthz/ready": read tcp 10.134.0.2:49056->10.134.0.134:15021: read: connection reset by peer... 41m Normal Started pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Container started 41m Normal ScalingReplicaSet deployment/e2e-inf-b-212d76-openshift-default Scaled up replica set e2e-inf-b-212d76-openshift-default-74c7cf55cc from 0 to 1 41m Normal AddedLabel gateway/e2e-inf-b-212d76 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-212d76 41m Normal SuccessfulCreate replicaset/e2e-inf-b-212d76-openshift-default-74c7cf55cc Created pod: e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb 41m Normal Created pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Container created 41m Normal Pulled pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Add eth0 [10.133.0.157/23] from ovn-kubernetes 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-5a2c68-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-5a2c68-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Normal Killing pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Stopping container payload-pre-processing 41m Normal Killing pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Stopping container payload-processing 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-212d76 Scaled up replica set payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff Error creating: pods "payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-212d76: serviceaccount "payload-processing-e2e-inf-b-212d76" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-212d76 Scaled up replica set payload-processing-e2e-inf-b-212d76-679cd8c679 from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-212d76-679cd8c679 Created pod: payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-212d76-679cd8c679 Error creating: pods "payload-processing-e2e-inf-b-212d76-679cd8c679-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-212d76: serviceaccount "payload-processing-e2e-inf-b-212d76" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff Created pod: payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g 41m Normal AddedInterface pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Add eth0 [10.134.0.150/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Container created 41m Normal Started pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Container started 41m Normal Pulled pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Container created 41m Normal Started pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Container started 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Add eth0 [10.133.0.158/23] from ovn-kubernetes 41m Warning ProbeError pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Startup probe error: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Startup probe failed: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-250992-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-250992-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-c68724-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-c68724-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-5a2c68-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-5a2c68-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning Unhealthy pod/payload-pre-processing-e2e-mt-47a126f3-5b7b569d67-m5tml Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Warning Unhealthy pod/payload-processing-e2e-mt-47a126f3-6f66b665c9-xbfkw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-250992-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-250992-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning Unhealthy pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Readiness probe failed: Get "http://10.134.0.142:15021/healthz/ready": read tcp 10.134.0.2:40316->10.134.0.142:15021: read: connection reset by peer 41m Warning ProbeError pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Readiness probe error: Get "http://10.134.0.142:15021/healthz/ready": read tcp 10.134.0.2:40316->10.134.0.142:15021: read: connection reset by peer... 41m Normal Pulled pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Pulled pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Container started 41m Normal Created pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Container created 41m Normal Created pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Container created 41m Normal Started pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Container started 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-1a771e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-1a771e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning ProbeError pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 40m Warning ProbeError pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Startup probe error: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Startup probe failed: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused 40m Warning Unhealthy pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 40m Normal AddedLabel gateway/e2e-inf-b-027d32 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-027d32 40m Normal ScalingReplicaSet deployment/e2e-inf-b-027d32-openshift-default Scaled up replica set e2e-inf-b-027d32-openshift-default-57d8dc6b5c from 0 to 1 40m Normal SuccessfulCreate replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Created pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj 40m Normal AddedInterface pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Add eth0 [10.133.0.161/23] from ovn-kubernetes 40m Normal AddedLabel gateway/e2e-api-b-6d4cd0 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-6d4cd0 40m Normal SuccessfulCreate replicaset/e2e-api-b-6d4cd0-openshift-default-5bdf489645 Created pod: e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x 40m Normal AddedInterface pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Add eth0 [10.133.0.162/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/e2e-api-b-6d4cd0-openshift-default Scaled up replica set e2e-api-b-6d4cd0-openshift-default-5bdf489645 from 0 to 1 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default New size: 4; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 40m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-dd8b6 Stopping container istio-proxy 40m Normal Created pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Container created 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-027d32 Scaled up replica set payload-pre-processing-e2e-inf-b-027d32-6f964b6967 from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-027d32 Scaled up replica set payload-processing-e2e-inf-b-027d32-585d5c5c5f from 0 to 1 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-027d32-585d5c5c5f Error creating: pods "payload-processing-e2e-inf-b-027d32-585d5c5c5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-027d32: serviceaccount "payload-processing-e2e-inf-b-027d32" not found 40m Normal Pulled pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Started pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Container started 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-027d32-585d5c5c5f Created pod: payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-027d32-6f964b6967 Error creating: pods "payload-pre-processing-e2e-inf-b-027d32-6f964b6967-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-027d32: serviceaccount "payload-processing-e2e-inf-b-027d32" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-027d32-6f964b6967 Created pod: payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm 40m Normal Pulled pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Started pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Container started 40m Normal Started pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Container started 40m Normal Created pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Container created 40m Normal Pulled pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Add eth0 [10.134.0.152/23] from ovn-kubernetes 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Add eth0 [10.134.0.153/23] from ovn-kubernetes 40m Normal Created pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Container created 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-b27079-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-6d4cd0 Scaled up replica set payload-processing-e2e-api-b-6d4cd0-667986d88c from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f Created pod: payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-6d4cd0 Scaled up replica set payload-pre-processing-e2e-api-b-6d4cd0-787d77979f from 0 to 1 40m Warning FailedCreate replicaset/payload-processing-e2e-api-b-6d4cd0-667986d88c Error creating: pods "payload-processing-e2e-api-b-6d4cd0-667986d88c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-6d4cd0: serviceaccount "payload-processing-e2e-api-b-6d4cd0" not found 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-b27079-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f Error creating: pods "payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-6d4cd0: serviceaccount "payload-processing-e2e-api-b-6d4cd0" not found 40m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Add eth0 [10.133.0.163/23] from ovn-kubernetes 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-6d4cd0-667986d88c Created pod: payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb 40m Normal Created pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Container created 40m Normal Pulled pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Started pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Container started 40m Normal Created pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Container created 40m Normal Started pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Container started 40m Normal Pulled pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Add eth0 [10.134.0.154/23] from ovn-kubernetes 40m Warning Unhealthy pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Startup probe failed: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused 40m Warning ProbeError pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Startup probe error: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused... 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-1a771e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-1a771e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal Killing pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Stopping container istio-proxy 40m Warning ProbeError pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-api-a-a37c29-openshift-default-77fdcb54f5-x7h6k Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Readiness probe error: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 40m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Readiness probe failed: Get "http://10.134.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-e1d6ff-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-e1d6ff-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 4; reason: All metrics below target 40m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled down replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 5 to 4 40m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-qw8q9 Stopping container istio-proxy 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 5; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-dsfsd 40m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled down replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 6 to 5 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-79pzd 40m Normal Killing pod/payload-pre-processing-e2e-api-b-250992-6d475684bc-g6cxh Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-api-b-250992-5d8ffd9995-7x6cb Stopping container payload-processing 40m Normal Pulled pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Container created 40m Normal Started pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Container started 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-212d76-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-212d76-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal Killing pod/payload-pre-processing-e2e-api-a-c68724-dbd5b895-4b7rb Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-api-a-c68724-697d45f96c-h5p76 Stopping container payload-processing 40m Warning ProbeError pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Startup probe error: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Startup probe failed: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-e1d6ff-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-e1d6ff-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 Stopping container istio-proxy 39m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default Scaled down replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 4 to 3 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 3; reason: All metrics below target 39m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-6qrw7 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-212d76-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-212d76-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Normal ScalingReplicaSet deployment/e2e-inf-b-027d32-openshift-default Scaled up replica set e2e-inf-b-027d32-openshift-default-57d8dc6b5c from 1 to 4 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 39m Normal SuccessfulCreate replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Created pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 39m Normal Pulled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Killing pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Stopping container istio-proxy 39m Normal SuccessfulCreate replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Created pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk 39m Normal Created pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Container created 39m Normal Pulled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Add eth0 [10.133.0.167/23] from ovn-kubernetes 39m Normal Started pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Container started 39m Normal Killing pod/e2e-mt-47a126f3-openshift-default-5789f86f77-vxv4j Stopping container istio-proxy 39m Normal Created pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Container created 39m Normal AddedInterface pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Add eth0 [10.134.0.158/23] from ovn-kubernetes 39m Normal Pulled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Add eth0 [10.132.0.120/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Created pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 39m Normal Started pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Container started 39m Normal Created pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Container created 39m Normal Started pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Container started 39m Normal Pulled pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Warning ProbeError pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Readiness probe error: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused... 39m Normal Created pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Container created 39m Normal Started pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Container started 39m Warning Unhealthy pod/e2e-mt-47a126f3-openshift-default-5789f86f77-tpw4s Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Readiness probe failed: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-6d4cd0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-6d4cd0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Pulled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Container created 39m Normal Started pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Container started 39m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Startup probe error: Get "http://10.134.0.158:15021/healthz/ready": dial tcp 10.134.0.158:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Startup probe failed: Get "http://10.133.0.167:15021/healthz/ready": dial tcp 10.133.0.167:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Startup probe failed: Get "http://10.134.0.158:15021/healthz/ready": dial tcp 10.134.0.158:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Startup probe error: Get "http://10.133.0.167:15021/healthz/ready": dial tcp 10.133.0.167:15021: connect: connection refused... 39m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Startup probe error: Get "http://10.132.0.120:15021/healthz/ready": dial tcp 10.132.0.120:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Startup probe failed: Get "http://10.132.0.120:15021/healthz/ready": dial tcp 10.132.0.120:15021: connect: connection refused 39m Warning Unhealthy pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Startup probe failed: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused 39m Warning ProbeError pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Startup probe error: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Startup probe failed: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused 39m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default Scaled down replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 5 to 3 39m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Startup probe error: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused... 39m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Stopping container istio-proxy 39m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 Stopping container istio-proxy 39m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 39m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-zkxn7 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 3; reason: All metrics below target 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 3; reason: All metrics below target 39m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default Scaled down replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 5 to 3 39m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled down replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 5 to 3 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 3; reason: All metrics below target 39m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k 39m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k 39m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv 39m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-xcn5k Stopping container istio-proxy 39m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-swmfv Stopping container istio-proxy 39m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh 39m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-mj56k Stopping container istio-proxy 39m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-k6jbh Stopping container istio-proxy 39m Normal SuccessfulCreate replicaset/e2e-inf-a-3d520d-openshift-default-5c79bbb455 Created pod: e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r 39m Normal AddedLabel gateway/e2e-inf-a-3d520d Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-3d520d 39m Normal ScalingReplicaSet deployment/e2e-inf-a-3d520d-openshift-default Scaled up replica set e2e-inf-a-3d520d-openshift-default-5c79bbb455 from 0 to 1 39m Normal AddedInterface pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Add eth0 [10.133.0.168/23] from ovn-kubernetes 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Readiness probe error: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-gjds4 Readiness probe failed: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default Scaled down replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 5 to 4 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 4; reason: All metrics below target 39m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 39m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-dz6j8 Stopping container istio-proxy 39m Normal Killing pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Stopping container istio-proxy 39m Normal Killing pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Stopping container istio-proxy 39m Normal Killing pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Stopping container istio-proxy 39m Normal Killing pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-97ccm Stopping container istio-proxy 39m Normal Killing pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Stopping container istio-proxy 39m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-hhrrr Readiness probe error: HTTP probe failed with statuscode: 503... 39m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Readiness probe failed: HTTP probe failed with statuscode: 503 39m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-chkq7 Readiness probe error: HTTP probe failed with statuscode: 503... 39m Normal SuccessfulCreate replicaset/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc Created pod: e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx 39m Normal ScalingReplicaSet deployment/e2e-inf-a-50caaa-openshift-default Scaled up replica set e2e-inf-a-50caaa-openshift-default-75b4ffb9cc from 0 to 1 39m Normal AddedLabel gateway/e2e-inf-a-50caaa Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-50caaa 38m Normal AddedInterface pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Add eth0 [10.134.0.159/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-3d520d-84969c4847 Created pod: payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-3d520d Scaled up replica set payload-pre-processing-e2e-inf-a-3d520d-84969c4847 from 0 to 1 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-3d520d-7dc58f4645 Error creating: pods "payload-processing-e2e-inf-a-3d520d-7dc58f4645-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-3d520d: serviceaccount "payload-processing-e2e-inf-a-3d520d" not found 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-3d520d-7dc58f4645 Created pod: payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-3d520d Scaled up replica set payload-processing-e2e-inf-a-3d520d-7dc58f4645 from 0 to 1 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-3d520d-84969c4847 Error creating: pods "payload-pre-processing-e2e-inf-a-3d520d-84969c4847-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-3d520d: serviceaccount "payload-processing-e2e-inf-a-3d520d" not found 38m Warning ProbeError pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning Unhealthy pod/e2e-api-a-c68724-openshift-default-6b8f8769bf-99rdj Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Normal Started pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Container started 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Add eth0 [10.133.0.171/23] from ovn-kubernetes 38m Normal Created pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Add eth0 [10.133.0.170/23] from ovn-kubernetes 38m Normal Pulled pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Container created 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Warning Unhealthy pod/e2e-api-b-250992-openshift-default-85d78c677c-h4md9 Readiness probe failed: HTTP probe failed with statuscode: 503 38m Normal Started pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Container started 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc Created pod: payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-50caaa Scaled up replica set payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc from 0 to 1 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc Error creating: pods "payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-50caaa: serviceaccount "payload-processing-e2e-inf-a-50caaa" not found 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-50caaa Scaled up replica set payload-processing-e2e-inf-a-50caaa-5fd5d5f699 from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-50caaa-5fd5d5f699 Created pod: payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-50caaa-5fd5d5f699 Error creating: pods "payload-processing-e2e-inf-a-50caaa-5fd5d5f699-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-50caaa: serviceaccount "payload-processing-e2e-inf-a-50caaa" not found 38m Normal Started pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Container started 38m Normal Created pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Add eth0 [10.134.0.160/23] from ovn-kubernetes 38m Normal Created pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Container created 38m Normal Pulled pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Add eth0 [10.134.0.161/23] from ovn-kubernetes 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Started pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Container started 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-6d4cd0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-6d4cd0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal Killing pod/payload-processing-e2e-api-b-e1d6ff-b5b9bfd85-w988v Stopping container payload-processing 38m Normal Killing pod/payload-pre-processing-e2e-api-b-e1d6ff-5f8f6d779-pr9gg Stopping container payload-pre-processing 38m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Readiness probe failed: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Readiness probe error: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-3d520d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-3d520d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal Started pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Container started 38m Normal Pulled pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Container created 38m Normal AddedInterface pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Add eth0 [10.134.0.163/23] from ovn-kubernetes 38m Normal ScalingReplicaSet deployment/e2e-inf-b-027d32-openshift-default Scaled up replica set e2e-inf-b-027d32-openshift-default-57d8dc6b5c from 4 to 5 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default New size: 5; reason: All metrics below target 38m Normal SuccessfulCreate replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Created pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd 38m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 Stopping container istio-proxy 38m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-fzwp2 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default New size: 3; reason: All metrics below target 38m Warning Unhealthy pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Startup probe failed: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Startup probe error: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused... 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-50caaa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-50caaa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal Killing pod/payload-pre-processing-e2e-api-a-1a771e-58879567c5-pd92j Stopping container payload-pre-processing 38m Normal Killing pod/payload-processing-e2e-api-a-1a771e-65d78995c5-lq6l2 Stopping container payload-processing 37m Normal Killing pod/payload-processing-e2e-shared-a-w3-25ff27-778dfd8567-82b6s Stopping container payload-processing 37m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-25ff27-74894c8b99-8jz2r Stopping container payload-pre-processing 37m Normal Started pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Container started 37m Normal Created pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Container created 37m Normal Pulled pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Container started 37m Normal Created pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Container created 37m Normal Pulled pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Warning ProbeError pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Readiness probe error: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Readiness probe failed: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused 37m Normal Killing pod/e2e-api-b-e1d6ff-openshift-default-5b9fbcd6c9-j2tcz Stopping container istio-proxy 37m Normal AddedLabel gateway/e2e-inf-a-0b0bd3 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-0b0bd3 37m Normal ScalingReplicaSet deployment/e2e-inf-a-0b0bd3-openshift-default Scaled up replica set e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c from 0 to 1 37m Warning Unhealthy pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Startup probe failed: Get "http://10.133.0.168:15021/healthz/ready": dial tcp 10.133.0.168:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Startup probe error: Get "http://10.133.0.168:15021/healthz/ready": dial tcp 10.133.0.168:15021: connect: connection refused... 37m Normal SuccessfulCreate replicaset/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c Created pod: e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv 37m Normal AddedInterface pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Add eth0 [10.134.0.172/23] from ovn-kubernetes 37m Warning Unhealthy pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Startup probe failed: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Startup probe error: Get "http://10.134.0.159:15021/healthz/ready": dial tcp 10.134.0.159:15021: connect: connection refused... 37m Normal Killing pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Stopping container istio-proxy 37m Normal ScalingReplicaSet deployment/e2e-inf-b-954b03-openshift-default Scaled up replica set e2e-inf-b-954b03-openshift-default-7b4665b955 from 0 to 1 37m Normal AddedLabel gateway/e2e-inf-b-954b03 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-954b03 37m Normal SuccessfulCreate replicaset/e2e-inf-b-954b03-openshift-default-7b4665b955 Created pod: e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w 37m Normal ScalingReplicaSet deployment/e2e-inf-b-dcb543-openshift-default Scaled up replica set e2e-inf-b-dcb543-openshift-default-7778f6dc5f from 0 to 1 37m Normal AddedLabel gateway/e2e-inf-b-dcb543 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-dcb543 37m Normal AddedInterface pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Add eth0 [10.133.0.190/23] from ovn-kubernetes 37m Normal AddedInterface pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Add eth0 [10.134.0.173/23] from ovn-kubernetes 37m Normal SuccessfulCreate replicaset/e2e-inf-b-dcb543-openshift-default-7778f6dc5f Created pod: e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr 37m Warning Unhealthy pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Readiness probe failed: HTTP probe failed with statuscode: 503 37m Warning ProbeError pod/e2e-api-a-1a771e-openshift-default-d4679577c-2sbjk Readiness probe error: HTTP probe failed with statuscode: 503... 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-dcb543 Scaled up replica set payload-pre-processing-e2e-inf-b-dcb543-576c4c7795 from 0 to 1 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-dcb543 Scaled up replica set payload-processing-e2e-inf-b-dcb543-7695df96b5 from 0 to 1 37m Normal Pulled pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795 Created pod: payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-dcb543-7695df96b5 Created pod: payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd 37m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Add eth0 [10.132.0.125/23] from ovn-kubernetes 37m Normal Started pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Container started 37m Normal Created pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Container created 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795 Error creating: pods "payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-dcb543: serviceaccount "payload-processing-e2e-inf-b-dcb543" not found 37m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-dcb543-7695df96b5 Error creating: pods "payload-processing-e2e-inf-b-dcb543-7695df96b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-dcb543: serviceaccount "payload-processing-e2e-inf-b-dcb543" not found 37m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87 Error creating: pods "payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0b0bd3: serviceaccount "payload-processing-e2e-inf-a-0b0bd3" not found 37m Normal Started pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Container started 37m Normal Pulled pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-0b0bd3 Scaled up replica set payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f from 0 to 1 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f Error creating: pods "payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-0b0bd3: serviceaccount "payload-processing-e2e-inf-a-0b0bd3" not found 37m Normal Created pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Container created 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87 Created pod: payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg 37m Normal AddedInterface pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Add eth0 [10.134.0.174/23] from ovn-kubernetes 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f Created pod: payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-0b0bd3 Scaled up replica set payload-processing-e2e-inf-a-0b0bd3-6b7848cb87 from 0 to 1 37m Normal Started pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Container started 37m Normal AddedInterface pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Add eth0 [10.132.0.126/23] from ovn-kubernetes 37m Normal Killing pod/payload-processing-e2e-api-b-6d4cd0-667986d88c-4vcbb Stopping container payload-processing 37m Normal Created pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Container created 37m Normal Killing pod/payload-pre-processing-e2e-api-b-6d4cd0-787d77979f-fvjz8 Stopping container payload-pre-processing 37m Normal Started pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Container started 37m Normal Created pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Container created 37m Normal Pulled pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal Pulled pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Add eth0 [10.134.0.175/23] from ovn-kubernetes 37m Normal Killing pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Stopping container istio-proxy 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-3d520d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-954b03 Scaled up replica set payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5 from 0 to 1 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-954b03-668f578fdf Error creating: pods "payload-processing-e2e-inf-b-954b03-668f578fdf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-954b03: serviceaccount "payload-processing-e2e-inf-b-954b03" not found 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-954b03 Scaled up replica set payload-processing-e2e-inf-b-954b03-668f578fdf from 0 to 1 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-3d520d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-954b03-668f578fdf Created pod: payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5 Error creating: pods "payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-954b03: serviceaccount "payload-processing-e2e-inf-b-954b03" not found 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5 Created pod: payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc 36m Normal Created pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Container created 36m Normal Pulled pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Add eth0 [10.134.0.176/23] from ovn-kubernetes 36m Normal Started pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Container started 36m Normal Created pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Container created 36m Normal Started pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Container started 36m Normal Pulled pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Add eth0 [10.134.0.177/23] from ovn-kubernetes 36m Warning ProbeError pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Readiness probe error: Get "http://10.134.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning Unhealthy pod/e2e-shared-a-w3-25ff27-openshift-default-856fc89989-rg6pr Readiness probe failed: Get "http://10.134.0.106:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Normal Killing pod/payload-processing-e2e-shared-b-w3-ba4687-7957fb899c-ljjrz Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-ba4687-789d455cf-r2699 Stopping container payload-pre-processing 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-50caaa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-50caaa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal ScalingReplicaSet deployment/e2e-worker-w1-7c933c-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-7c933c-openshift-default-6d45577bf5 from 3 to 2 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-7c933c-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/e2e-worker-w1-7c933c-openshift-default-6d45577bf5 Deleted pod: e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj 36m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Stopping container istio-proxy 36m Warning Unhealthy pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-j2bvj Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/e2e-shared-b-w3-ba4687-openshift-default-5c9c79f64c-rxcrb Stopping container istio-proxy 35m Normal Killing pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Stopping container payload-pre-processing 35m Warning ProbeError pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Readiness probe error: Get "http://10.134.0.172:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 35m Warning Unhealthy pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Readiness probe failed: Get "http://10.134.0.172:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 35m Normal Started pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Container started 35m Normal Pulled pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Container created 35m Normal Started pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Container started 35m Normal Created pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Container created 35m Normal Pulled pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Container created 35m Normal Started pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Container started 35m Normal Pulled pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-dcb543-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-dcb543-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal ScalingReplicaSet deployment/e2e-inf-b-ab76d3-openshift-default Scaled up replica set e2e-inf-b-ab76d3-openshift-default-6755f6cd4c from 0 to 1 35m Normal AddedLabel gateway/e2e-inf-b-ab76d3 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-ab76d3 35m Normal SuccessfulCreate replicaset/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c Created pod: e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz 35m Normal AddedInterface pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Add eth0 [10.134.0.178/23] from ovn-kubernetes 35m Warning Unhealthy pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Startup probe failed: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused 35m Warning ProbeError pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Startup probe error: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused... 35m Warning ProbeError pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Startup probe error: Get "http://10.134.0.173:15021/healthz/ready": dial tcp 10.134.0.173:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Startup probe failed: Get "http://10.134.0.173:15021/healthz/ready": dial tcp 10.134.0.173:15021: connect: connection refused 35m Warning ProbeError pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Startup probe error: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Startup probe failed: Get "http://10.134.0.172:15021/healthz/ready": dial tcp 10.134.0.172:15021: connect: connection refused 35m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-159cb7-66bb4fdd79-nsczv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-processing-e2e-worker-w3-159cb7-cc54b644c-vkdnn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5 Error creating: pods "payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-ab76d3: serviceaccount "payload-processing-e2e-inf-b-ab76d3" not found 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c Created pod: payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-ab76d3 Scaled up replica set payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c from 0 to 1 35m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c Error creating: pods "payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-ab76d3: serviceaccount "payload-processing-e2e-inf-b-ab76d3" not found 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-ab76d3 Scaled up replica set payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5 from 0 to 1 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5 Created pod: payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz 35m Normal Created pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Container created 35m Normal Pulled pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Started pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Container started 35m Normal Created pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Container created 35m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Add eth0 [10.134.0.180/23] from ovn-kubernetes 35m Normal AddedInterface pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Add eth0 [10.134.0.179/23] from ovn-kubernetes 35m Normal Pulled pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal Started pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Container started 35m Normal Killing pod/e2e-api-b-6d4cd0-openshift-default-5bdf489645-wsn5x Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/e2e-inf-a-579ea7-openshift-default Scaled up replica set e2e-inf-a-579ea7-openshift-default-86c8f5c95 from 0 to 1 35m Normal SuccessfulCreate replicaset/e2e-inf-a-579ea7-openshift-default-86c8f5c95 Created pod: e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t 35m Normal AddedLabel gateway/e2e-inf-a-579ea7 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-579ea7 35m Normal AddedInterface pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Add eth0 [10.134.0.181/23] from ovn-kubernetes 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-954b03-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-954b03-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-ab76d3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-ab76d3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal Pulled pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-579ea7 Scaled up replica set payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f from 0 to 1 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-0b0bd3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Add eth0 [10.133.0.198/23] from ovn-kubernetes 35m Normal Pulled pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 35m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-579ea7-5489bc9fdf Created pod: payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 35m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-579ea7 Scaled up replica set payload-processing-e2e-inf-a-579ea7-5489bc9fdf from 0 to 1 35m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-579ea7-5489bc9fdf Error creating: pods "payload-processing-e2e-inf-a-579ea7-5489bc9fdf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-579ea7: serviceaccount "payload-processing-e2e-inf-a-579ea7" not found 35m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f Error creating: pods "payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-579ea7: serviceaccount "payload-processing-e2e-inf-a-579ea7" not found 35m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f Created pod: payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms 35m Normal AddedInterface pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Add eth0 [10.133.0.197/23] from ovn-kubernetes 35m Normal Created pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Container created 35m Normal Started pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Container started 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-0b0bd3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Normal Created pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Container created 35m Normal Started pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Container started 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-579ea7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-579ea7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-954b03-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-954b03-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal ScalingReplicaSet deployment/e2e-worker-w3-159cb7-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w3-159cb7-openshift-default-6b9cb98757 from 3 to 2 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-159cb7-openshift-default New size: 2; reason: All metrics below target 35m Normal SuccessfulDelete replicaset/e2e-worker-w3-159cb7-openshift-default-6b9cb98757 Deleted pod: e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb 35m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-kqkmb Stopping container istio-proxy 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-dcb543-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-dcb543-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-5wz42 Stopping container istio-proxy 34m Normal Killing pod/e2e-worker-w3-159cb7-openshift-default-6b9cb98757-sg5s7 Stopping container istio-proxy 34m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg 34m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled down replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 3 to 2 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 2; reason: All metrics below target 34m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-4nghg Stopping container istio-proxy 34m Normal Killing pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Stopping container payload-processing 34m Normal Killing pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Stopping container payload-pre-processing 34m Normal SuccessfulDelete replicaset/e2e-inf-b-027d32-openshift-default-57d8dc6b5c Deleted pod: e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd 34m Normal ScalingReplicaSet deployment/e2e-inf-b-027d32-openshift-default Scaled down replica set e2e-inf-b-027d32-openshift-default-57d8dc6b5c from 5 to 4 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-027d32-openshift-default New size: 4; reason: All metrics below target 34m Normal Killing pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-2fnjd Stopping container istio-proxy 34m Normal Created pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Container created 34m Normal Pulled pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Container started 34m Normal Pulled pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Container created 34m Normal Started pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Container started 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-5a2c68-5f89d5d78c-8h9qq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning Unhealthy pod/payload-processing-e2e-inf-a-5a2c68-59b9485d87-7t89m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning Unhealthy pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Startup probe failed: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Startup probe error: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused... 34m Warning ProbeError pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Startup probe error: Get "http://10.134.0.178:15021/healthz/ready": dial tcp 10.134.0.178:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Startup probe failed: Get "http://10.134.0.178:15021/healthz/ready": dial tcp 10.134.0.178:15021: connect: connection refused 33m Normal SuccessfulCreate replicaset/e2e-inf-b-58ac21-openshift-default-75d766fb8f Created pod: e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-dcb543-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 33m Normal ScalingReplicaSet deployment/e2e-inf-b-dcb543-openshift-default Scaled up replica set e2e-inf-b-dcb543-openshift-default-7778f6dc5f from 1 to 2 33m Normal SuccessfulCreate replicaset/e2e-inf-b-dcb543-openshift-default-7778f6dc5f Created pod: e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 33m Normal AddedLabel gateway/e2e-inf-b-58ac21 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-58ac21 33m Normal ScalingReplicaSet deployment/e2e-inf-b-58ac21-openshift-default Scaled up replica set e2e-inf-b-58ac21-openshift-default-75d766fb8f from 0 to 1 33m Normal AddedInterface pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Add eth0 [10.134.0.186/23] from ovn-kubernetes 33m Normal Started pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Container started 33m Normal Created pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Container created 33m Normal Pulled pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Add eth0 [10.134.0.185/23] from ovn-kubernetes 33m Normal Killing pod/e2e-inf-a-5a2c68-openshift-default-5db97fc557-fhr7j Stopping container istio-proxy 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-58ac21 Scaled up replica set payload-pre-processing-e2e-inf-b-58ac21-575576949f from 0 to 1 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-58ac21-575576949f Error creating: pods "payload-pre-processing-e2e-inf-b-58ac21-575576949f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-58ac21: serviceaccount "payload-processing-e2e-inf-b-58ac21" not found 33m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9 Error creating: pods "payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-58ac21: serviceaccount "payload-processing-e2e-inf-b-58ac21" not found 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9 Created pod: payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-58ac21 Scaled up replica set payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9 from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-58ac21-575576949f Created pod: payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 33m Normal Pulled pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Container created 33m Warning Unhealthy pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Startup probe failed: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused 33m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Add eth0 [10.133.0.204/23] from ovn-kubernetes 33m Normal AddedInterface pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Add eth0 [10.134.0.187/23] from ovn-kubernetes 33m Warning ProbeError pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Startup probe error: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused... 33m Normal Started pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Container started 33m Normal Created pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Container created 33m Normal Started pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Container started 33m Normal Pulled pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-579ea7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Normal Killing pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Stopping container payload-pre-processing 33m Normal Killing pod/payload-processing-e2e-inf-b-212d76-679cd8c679-xs574 Stopping container payload-processing 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-579ea7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-ab76d3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-ab76d3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-212d76-685cdcc9ff-2bs6g Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal Killing pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Stopping container istio-proxy 33m Warning ProbeError pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/e2e-inf-b-212d76-openshift-default-74c7cf55cc-tgqjb Readiness probe failed: HTTP probe failed with statuscode: 503 33m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-e2dfd6-69b5599dcf-4xsj8 Stopping container payload-pre-processing 33m Normal Killing pod/payload-processing-e2e-shared-a-w1-e2dfd6-7d7b5d94f-vxnzz Stopping container payload-processing 32m Normal Killing pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Stopping container payload-pre-processing 32m Normal Pulled pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Created pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Container created 32m Normal Started pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Container started 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-b27079-64599b455-sv4dm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-processing-e2e-inf-a-b27079-bd68fb84c-qtspk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning ProbeError pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Startup probe error: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Startup probe failed: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused 32m Normal Killing pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Stopping container istio-proxy 32m Warning Unhealthy pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Readiness probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 32m Warning ProbeError pod/e2e-inf-a-b27079-openshift-default-5f79bb8845-zh8x2 Readiness probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-58ac21-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-58ac21-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Stopping container payload-pre-processing 32m Normal Killing pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Stopping container payload-processing 32m Normal Killing pod/e2e-shared-a-w1-e2dfd6-openshift-default-dd5bc48f9-wblpz Stopping container istio-proxy 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-58ac21-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-58ac21-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Normal Killing pod/payload-processing-e2e-shared-b-w1-a5faaa-79c69b4c5-sd7n9 Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-a5faaa-669c6fb6dc-7xdsv Stopping container payload-pre-processing 31m Warning Unhealthy pod/payload-processing-e2e-inf-b-027d32-585d5c5c5f-b2h5n Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-027d32-6f964b6967-gsgqm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-h4wkx 31m Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-rdnfs Stopping container istio-proxy 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-rdnfs 31m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-79dq2 31m Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-79dq2 Stopping container istio-proxy 31m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Readiness probe error: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 31m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-h4wkx Readiness probe failed: Get "http://10.134.0.61:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 31m Normal Killing pod/e2e-shared-b-w1-a5faaa-openshift-default-c59d68c49-h4rnw Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-lpnnk Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-scq66 Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-dkdj5 Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-027d32-openshift-default-57d8dc6b5c-4pxsj Stopping container istio-proxy 31m Normal Killing pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Stopping container payload-processing 31m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-c839e5-687464fb99-2bxxc Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-shared-a-w0-c839e5-78bd9f46c9-7pm9f Stopping container payload-processing 31m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-7c933c-64794cc4d7-ztk96 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Warning Unhealthy pod/payload-processing-e2e-worker-w1-7c933c-5b84567d97-6sd52 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal Killing pod/e2e-shared-a-w0-c839e5-openshift-default-6c446cf677-577tx Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-vpnvf Stopping container istio-proxy 30m Normal Killing pod/e2e-worker-w1-7c933c-openshift-default-6d45577bf5-8nq7n Stopping container istio-proxy 30m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-e69af1-59d6f47bdc-rfd8v Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-shared-b-w0-e69af1-755b48899f-vlzxp Stopping container payload-processing 30m Normal Killing pod/e2e-shared-b-w0-e69af1-openshift-default-76f9f7f7c7-lr6q6 Stopping container istio-proxy 30m Normal Killing pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Stopping container payload-pre-processing 30m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-d76xz 30m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 30m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 30m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-d76xz Stopping container discovery 30m Normal ScalingReplicaSet deployment/e2e-worker-w2-1fb1c6-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 from 3 to 2 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-1fb1c6-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 30m Normal SuccessfulDelete replicaset/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7 Deleted pod: e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw 30m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Stopping container istio-proxy 30m Warning Unhealthy pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning ProbeError pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-nqrmw Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/payload-processing-e2e-worker-w0-e1a7f5-7859859c4f-sfqbx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-5d73b9-openshift-default New size: 2; reason: All metrics below target 29m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w0-e1a7f5-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 from 3 to 2 29m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-e1a7f5-755bdc95ff-b6g5m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-e1a7f5-openshift-default New size: 2; reason: All metrics below target 29m Normal SuccessfulDelete replicaset/e2e-worker-w6-5d73b9-openshift-default-fb75d798f Deleted pod: e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs 29m Normal SuccessfulDelete replicaset/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9 Deleted pod: e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 29m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w6-5d73b9-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-5d73b9-openshift-default-fb75d798f from 3 to 2 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-bb48ef-openshift-default New size: 1; reason: All metrics below target 29m Normal ScalingReplicaSet deployment/e2e-worker-w4-bb48ef-openshift-default Scaled down replica set e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 from 2 to 1 29m Normal SuccessfulDelete replicaset/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9 Deleted pod: e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf 29m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-pq8cf Stopping container istio-proxy 29m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-m4552 29m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-cvtfw 29m Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-cvtfw Stopping container istio-proxy 29m Warning Unhealthy pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Readiness probe failed: HTTP probe failed with statuscode: 503 29m Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Stopping container istio-proxy 29m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 29m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-9dccdc9b7 from 7 to 5 29m Warning ProbeError pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-ktr46 Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning ProbeError pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-lqrfs Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-m4552 Readiness probe error: HTTP probe failed with statuscode: 503... 29m Normal SuccessfulDelete replicaset/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 Deleted pod: e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws 29m Normal Killing pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-worker-w5-e2e897-openshift-default Scaled down replica set e2e-worker-w5-e2e897-openshift-default-9bbb9fc59 from 2 to 1 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-e2e897-openshift-default New size: 1; reason: All metrics below target 29m Warning Unhealthy pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Readiness probe failed: HTTP probe failed with statuscode: 503 29m Warning ProbeError pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-ww2ws Readiness probe error: HTTP probe failed with statuscode: 503... 29m Normal Killing pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-inf-a-50caaa-5fd5d5f699-cgr28 Stopping container payload-processing 29m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-5qd6m Stopping container istio-proxy 29m Normal Killing pod/e2e-worker-w0-e1a7f5-openshift-default-c77858bf9-dsmwn Stopping container istio-proxy 29m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-50caaa-7bd5b4bbc-w8d79 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal Killing pod/e2e-inf-a-50caaa-openshift-default-75b4ffb9cc-4hkxx Stopping container istio-proxy 29m Normal Killing pod/payload-processing-e2e-inf-a-3d520d-7dc58f4645-fkpl9 Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Stopping container payload-pre-processing 28m Normal Killing pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Stopping container payload-pre-processing 28m Normal Killing pod/payload-processing-e2e-inf-b-954b03-668f578fdf-vvb57 Stopping container payload-processing 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-3d520d-84969c4847-7vtfq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal Killing pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Stopping container istio-proxy 28m Warning ProbeError pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Readiness probe error: HTTP probe failed with statuscode: 503... 28m Warning Unhealthy pod/e2e-inf-a-3d520d-openshift-default-5c79bbb455-h624r Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-954b03-5c9c69f4c5-wq6qc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal Killing pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Stopping container payload-pre-processing 28m Normal Killing pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Stopping container istio-proxy 28m Warning Unhealthy pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/e2e-inf-b-954b03-openshift-default-7b4665b955-64s8w Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal Killing pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Stopping container payload-pre-processing 28m Normal Killing pod/payload-processing-e2e-inf-a-0b0bd3-6b7848cb87-294zg Stopping container payload-processing 28m Warning Unhealthy pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Readiness probe failed: HTTP probe failed with statuscode: 503 28m Normal Killing pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-84tfr Stopping container istio-proxy 28m Warning ProbeError pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal Killing pod/e2e-inf-b-dcb543-openshift-default-7778f6dc5f-sq7j6 Stopping container istio-proxy 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-dcb543-576c4c7795-g8p2b Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Normal Killing pod/payload-processing-e2e-shared-a-w6-83a98b-59ff68f55f-zbqwk Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-83a98b-658b4b79f9-jpcl6 Stopping container payload-pre-processing 28m Normal Killing pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Stopping container payload-pre-processing 28m Warning Unhealthy pod/payload-processing-e2e-inf-b-dcb543-7695df96b5-dtvjd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-0b0bd3-d58557c5f-j4xtt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-processing-e2e-worker-w5-e2e897-75cbd44db9-hzqpt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-e2e897-7649d5c5bc-ffhf6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Stopping container istio-proxy 27m Warning ProbeError pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/e2e-inf-a-0b0bd3-openshift-default-65b5fdd89c-dpcmv Readiness probe failed: HTTP probe failed with statuscode: 503 27m Normal Killing pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-inf-a-579ea7-5489bc9fdf-d6fj4 Stopping container payload-processing 27m Normal Killing pod/e2e-shared-a-w6-83a98b-openshift-default-6bf744db5c-8qcbs Stopping container istio-proxy 27m Normal Killing pod/payload-processing-e2e-inf-b-ab76d3-5c6bd7fb5c-2xg66 Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Stopping container payload-pre-processing 27m Normal Killing pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Stopping container istio-proxy 27m Normal Killing pod/payload-processing-e2e-shared-b-w6-1d454b-6d8bc6b8b7-tdztr Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-1d454b-5cc668d4d7-tmr4q Stopping container payload-pre-processing 27m Warning Unhealthy pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning ProbeError pod/e2e-inf-a-579ea7-openshift-default-86c8f5c95-lnl5t Readiness probe error: HTTP probe failed with statuscode: 503... 27m Normal Killing pod/e2e-worker-w5-e2e897-openshift-default-9bbb9fc59-2pr4t Stopping container istio-proxy 27m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-579ea7-578c4b5f5f-kmmms Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/payload-processing-e2e-inf-b-58ac21-6f9bcdcdc9-nzbjg Stopping container payload-processing 27m Normal Killing pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Stopping container payload-pre-processing 27m Normal Killing pod/e2e-shared-b-w6-1d454b-openshift-default-6f6cdcffb5-nd54g Stopping container istio-proxy 27m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-ab76d3-7bb5bd98f5-2f5nz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Stopping container istio-proxy 27m Warning ProbeError pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning ProbeError pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Readiness probe error: HTTP probe failed with statuscode: 503... 27m Warning Unhealthy pod/e2e-inf-b-ab76d3-openshift-default-6755f6cd4c-6q4lz Readiness probe failed: HTTP probe failed with statuscode: 503 27m Warning Unhealthy pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Readiness probe failed: HTTP probe failed with statuscode: 503 27m Normal Killing pod/e2e-inf-b-58ac21-openshift-default-75d766fb8f-zzqcn Stopping container istio-proxy 27m Normal Killing pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Stopping container payload-processing 27m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-58ac21-575576949f-bzg49 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 27m Normal Killing pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Stopping container payload-processing 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-5d73b9-58b4f8667-2smd2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-processing-e2e-worker-w6-5d73b9-55b9cb97b7-hxmpv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Normal Killing pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Stopping container payload-processing 26m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-2g82z Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w6-5d73b9-openshift-default-fb75d798f-mj79m Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w4-bb48ef-openshift-default-85fb7c67f9-h48np Stopping container istio-proxy 26m Warning Unhealthy pod/payload-processing-e2e-worker-w4-bb48ef-7f4f99b669-pgvkk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-bb48ef-75fdb895f-lzlcv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-processing-e2e-worker-w2-1fb1c6-69b567b565-wm2ml Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-1fb1c6-5c7b8cc6f5-628zl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 26m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-7qtnp Stopping container istio-proxy 26m Normal Killing pod/e2e-worker-w2-1fb1c6-openshift-default-b6dd95fd7-rb9md Stopping container istio-proxy 26m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-9dccdc9b7 from 5 to 10 26m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 26m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-cf442 26m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-d8glx 26m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Add eth0 [10.134.0.197/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-lshsj 26m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-9gb6c 26m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Created pod: maas-default-gateway-openshift-default-9dccdc9b7-2msfs 26m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Container created 26m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Container created 26m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Container started 26m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Add eth0 [10.132.0.132/23] from ovn-kubernetes 26m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Container started 26m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Add eth0 [10.134.0.199/23] from ovn-kubernetes 26m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Container created 26m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Container started 26m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Container started 26m Normal Started pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Container started 26m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Container created 26m Normal Created pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Container created 26m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Add eth0 [10.134.0.198/23] from ovn-kubernetes 26m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Pulled pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Add eth0 [10.132.0.131/23] from ovn-kubernetes 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Startup probe failed: Get "http://10.132.0.131:15021/healthz/ready": dial tcp 10.132.0.131:15021: connect: connection refused 26m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Startup probe error: Get "http://10.132.0.132:15021/healthz/ready": dial tcp 10.132.0.132:15021: connect: connection refused... 26m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Startup probe error: Get "http://10.132.0.131:15021/healthz/ready": dial tcp 10.132.0.131:15021: connect: connection refused... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Startup probe failed: Get "http://10.132.0.132:15021/healthz/ready": dial tcp 10.132.0.132:15021: connect: connection refused 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Startup probe failed: Get "http://10.134.0.199:15021/healthz/ready": dial tcp 10.134.0.199:15021: connect: connection refused 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Startup probe failed: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused 26m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Startup probe error: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused... 26m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Startup probe error: Get "http://10.134.0.199:15021/healthz/ready": dial tcp 10.134.0.199:15021: connect: connection refused... 26m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Startup probe failed: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused 26m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Startup probe error: Get "http://10.134.0.197:15021/healthz/ready": dial tcp 10.134.0.197:15021: connect: connection refused... 25m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 25m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-lg2gf 24m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-lg2gf Stopping container discovery 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dmaas-default-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dmaas-default-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dopenshift-ai-inference\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dopenshift-ai-inference\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Distiod%2Cistio.io%2Frev%3Dopenshift-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Distiod%2Cistio.io%2Frev%3Dopenshift-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Dkube-auth-proxy\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Dkube-auth-proxy\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Ddata-science-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Ddata-science-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Dkube-auth-proxy\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Distiod%2Cistio.io%2Frev%3Dopenshift-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Distiod%2Cistio.io%2Frev%3Dopenshift-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dmaas-default-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=app%3Dkube-auth-proxy\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dopenshift-ai-inference\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dmaas-default-gateway\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": context deadline exceeded") has prevented the request from succeeding (get pods.metrics.k8s.io) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/openshift-ingress/pods?labelSelector=gateway.networking.k8s.io%2Fgateway-name%3Dopenshift-ai-inference\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) 18m Normal AddedLabel gateway/e2e-worker-single-main-c51413 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-c51413 18m Normal AddedInterface pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Add eth0 [10.134.0.217/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/e2e-worker-single-main-c51413-openshift-default-64f589b645 Created pod: e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw 18m Normal ScalingReplicaSet deployment/e2e-worker-single-main-c51413-openshift-default Scaled up replica set e2e-worker-single-main-c51413-openshift-default-64f589b645 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-c51413-558f487999 Created pod: payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-c51413 Scaled up replica set payload-pre-processing-e2e-worker-single-main-c51413-558f487999 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-c51413-558f487999 Error creating: pods "payload-pre-processing-e2e-worker-single-main-c51413-558f487999-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-c51413: serviceaccount "payload-processing-e2e-worker-single-main-c51413" not found 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-c51413 Scaled up replica set payload-processing-e2e-worker-single-main-c51413-574f54c99c from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-c51413-574f54c99c Created pod: payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 18m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-c51413-574f54c99c Error creating: pods "payload-processing-e2e-worker-single-main-c51413-574f54c99c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-c51413: serviceaccount "payload-processing-e2e-worker-single-main-c51413" not found 18m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Add eth0 [10.134.0.218/23] from ovn-kubernetes 18m Normal Created pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Container created 18m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Add eth0 [10.134.0.219/23] from ovn-kubernetes 18m Normal Started pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Container started 18m Normal Created pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Container created 18m Normal Pulled pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 18m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 18m Normal Started pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Container started 18m Normal Pulled pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Warning Unhealthy pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Readiness probe failed: Get "http://10.134.0.217:15021/healthz/ready": read tcp 10.134.0.2:37374->10.134.0.217:15021: read: connection reset by peer 18m Warning ProbeError pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Readiness probe error: Get "http://10.134.0.217:15021/healthz/ready": read tcp 10.134.0.2:37374->10.134.0.217:15021: read: connection reset by peer... 18m Normal Started pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Container started 18m Normal Created pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Container created 18m Warning Unhealthy pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Startup probe failed: Get "http://10.134.0.217:15021/healthz/ready": dial tcp 10.134.0.217:15021: connect: connection refused 18m Warning ProbeError pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Startup probe error: Get "http://10.134.0.217:15021/healthz/ready": dial tcp 10.134.0.217:15021: connect: connection refused... 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-c51413-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-c51413-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-c51413-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-c51413-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Normal Killing pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Stopping container payload-pre-processing 17m Normal Killing pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Stopping container istio-proxy 17m Warning ProbeError pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning Unhealthy pod/e2e-worker-single-main-c51413-openshift-default-64f589b645k5xzw Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-c51413-574f54c995wq59 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-c51413-558f4959lb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal AddedLabel gateway/e2e-worker-single-main-88d1c3 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-88d1c3 14m Normal ScalingReplicaSet deployment/e2e-worker-single-main-88d1c3-openshift-default Scaled up replica set e2e-worker-single-main-88d1c3-openshift-default-84b9c78989 from 0 to 1 14m Normal SuccessfulCreate replicaset/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989 Created pod: e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w 14m Normal AddedInterface pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Add eth0 [10.134.0.225/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7c Created pod: payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-88d1c3 Scaled up replica set payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd6887 from 0 to 1 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd6887 Error creating: pods "payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd6887-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-88d1c3: serviceaccount "payload-processing-e2e-worker-single-main-88d1c3" not found 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd6887 Created pod: payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7c Error creating: pods "payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-88d1c3: serviceaccount "payload-processing-e2e-worker-single-main-88d1c3" not found 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-88d1c3 Scaled up replica set payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7c from 0 to 1 13m Normal Created pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Container created 13m Normal Pulled pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Container created 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Add eth0 [10.134.0.227/23] from ovn-kubernetes 13m Normal Started pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Container started 13m Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Started pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Container started 13m Normal AddedInterface pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Add eth0 [10.134.0.226/23] from ovn-kubernetes 13m Normal Pulled pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Container created 13m Normal Started pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Container started 13m Warning Unhealthy pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Startup probe failed: Get "http://10.134.0.225:15021/healthz/ready": dial tcp 10.134.0.225:15021: connect: connection refused 13m Warning ProbeError pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Startup probe error: Get "http://10.134.0.225:15021/healthz/ready": dial tcp 10.134.0.225:15021: connect: connection refused... 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-88d1c3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-88d1c3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Normal Killing pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Stopping container payload-processing 12m Normal Killing pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Stopping container payload-pre-processing 12m Normal Killing pod/e2e-worker-single-main-88d1c3-openshift-default-84b9c78989r7z4w Stopping container istio-proxy 12m Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-88d1c3-c55fd7bbw7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 12m Warning Unhealthy pod/payload-processing-e2e-worker-single-main-88d1c3-68ffdd8b7vmxbp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Readiness probe failed: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Readiness probe error: Get "http://10.133.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Readiness probe failed: Get "http://10.133.0.56:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Readiness probe error: Get "http://10.132.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Readiness probe failed: Get "http://10.133.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Readiness probe error: Get "http://10.133.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 10m Normal AddedLabel gateway/e2e-ait-77f48536 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-77f48536 10m Normal AddedInterface pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Add eth0 [10.134.0.234/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/e2e-ait-77f48536-openshift-default-5cd488889c Created pod: e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf 10m Normal ScalingReplicaSet deployment/e2e-ait-77f48536-openshift-default Scaled up replica set e2e-ait-77f48536-openshift-default-5cd488889c from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-77f48536 Scaled up replica set payload-pre-processing-e2e-ait-77f48536-7579dfd599 from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-77f48536-7579dfd599 Created pod: payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-77f48536-6bb598695 Created pod: payload-processing-e2e-ait-77f48536-6bb598695-2ddkx 10m Warning FailedCreate replicaset/payload-processing-e2e-ait-77f48536-6bb598695 Error creating: pods "payload-processing-e2e-ait-77f48536-6bb598695-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-77f48536: serviceaccount "payload-processing-e2e-ait-77f48536" not found 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-77f48536 Scaled up replica set payload-processing-e2e-ait-77f48536-6bb598695 from 0 to 1 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-77f48536-7579dfd599 Error creating: pods "payload-pre-processing-e2e-ait-77f48536-7579dfd599-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-77f48536: serviceaccount "payload-processing-e2e-ait-77f48536" not found 10m Normal Pulled pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Container started 10m Normal Created pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Container created 10m Normal AddedInterface pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Add eth0 [10.132.0.143/23] from ovn-kubernetes 10m Normal Created pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Container created 10m Normal Pulled pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal AddedInterface pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Add eth0 [10.132.0.142/23] from ovn-kubernetes 10m Normal Started pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Container started 10m Warning ProbeError pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Readiness probe error: Get "http://10.134.0.234:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 10m Warning Unhealthy pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Readiness probe failed: Get "http://10.134.0.234:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 10m Normal Pulled pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Normal Created pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Container created 10m Normal Started pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Container started 10m Warning Unhealthy pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Startup probe failed: Get "http://10.134.0.234:15021/healthz/ready": dial tcp 10.134.0.234:15021: connect: connection refused 10m Warning ProbeError pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Startup probe error: Get "http://10.134.0.234:15021/healthz/ready": dial tcp 10.134.0.234:15021: connect: connection refused... 9m57s Normal Killing pod/payload-pre-processing-e2e-ait-77f48536-7579dfd599-28dwl Stopping container payload-pre-processing 9m57s Normal Killing pod/payload-processing-e2e-ait-77f48536-6bb598695-2ddkx Stopping container payload-processing 9m56s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-77f48536-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m56s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-77f48536-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m51s Normal Killing pod/e2e-ait-77f48536-openshift-default-5cd488889c-jb8lf Stopping container istio-proxy 9m49s Normal AddedLabel gateway/e2e-authcfg-http-95ffa8 Added label istio.io/rev=openshift-gateway to gateway e2e-authcfg-http-95ffa8 9m49s Normal SuccessfulCreate replicaset/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89 Created pod: e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv 9m49s Normal ScalingReplicaSet deployment/e2e-authcfg-http-95ffa8-openshift-default Scaled up replica set e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89 from 0 to 1 9m48s Normal AddedInterface pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Add eth0 [10.134.0.236/23] from ovn-kubernetes 9m43s Warning FailedCreate replicaset/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9 Error creating: pods "payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-95ffa8: serviceaccount "payload-processing-e2e-authcfg-http-95ffa8" not found 9m43s Warning FailedCreate replicaset/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7 Error creating: pods "payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-authcfg-http-95ffa8: serviceaccount "payload-processing-e2e-authcfg-http-95ffa8" not found 9m43s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7 Created pod: payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t 9m43s Normal AddedInterface pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Add eth0 [10.134.0.238/23] from ovn-kubernetes 9m43s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-authcfg-http-95ffa8 Scaled up replica set payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7 from 0 to 1 9m43s Normal ScalingReplicaSet deployment/payload-processing-e2e-authcfg-http-95ffa8 Scaled up replica set payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9 from 0 to 1 9m43s Normal AddedInterface pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Add eth0 [10.134.0.237/23] from ovn-kubernetes 9m43s Normal Pulled pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m43s Normal Created pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Container created 9m43s Normal SuccessfulCreate replicaset/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9 Created pod: payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h 9m42s Normal Pulled pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m42s Normal Started pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Container started 9m42s Normal Started pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Container started 9m42s Normal Created pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Container created 9m31s Warning ProbeError pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Readiness probe error: Get "http://10.134.0.236:15021/healthz/ready": read tcp 10.134.0.2:59612->10.134.0.236:15021: read: connection reset by peer... 9m31s Warning Unhealthy pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Readiness probe failed: Get "http://10.134.0.236:15021/healthz/ready": read tcp 10.134.0.2:59612->10.134.0.236:15021: read: connection reset by peer 9m30s Normal Started pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Container started 9m30s Normal Created pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Container created 9m30s Normal Pulled pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m25s Warning ProbeError pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Startup probe error: Get "http://10.134.0.236:15021/healthz/ready": dial tcp 10.134.0.236:15021: connect: connection refused... 9m25s Warning Unhealthy pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Startup probe failed: Get "http://10.134.0.236:15021/healthz/ready": dial tcp 10.134.0.236:15021: connect: connection refused 8m49s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-authcfg-http-95ffa8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m49s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-authcfg-http-95ffa8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m59s Normal Killing pod/payload-pre-processing-e2e-authcfg-http-95ffa8-75864498d7-q7c8t Stopping container payload-pre-processing 7m59s Normal Killing pod/payload-processing-e2e-authcfg-http-95ffa8-59b7b768b9-cpl2h Stopping container payload-processing 7m45s Normal Killing pod/e2e-authcfg-http-95ffa8-openshift-default-7594d7cc89-652lv Stopping container istio-proxy 7m30s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Readiness probe error: Get "http://10.134.0.198:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m30s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Readiness probe failed: Get "http://10.134.0.197:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m30s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Readiness probe error: Get "http://10.134.0.197:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m30s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Readiness probe error: Get "http://10.134.0.199:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m30s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Readiness probe failed: Get "http://10.134.0.199:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m30s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Readiness probe failed: Get "http://10.134.0.198:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 7m1s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Readiness probe error: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 7m1s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Readiness probe failed: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m36s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Readiness probe failed: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m36s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-vvv67 Readiness probe error: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 79s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-9gb6c 79s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 79s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-9dccdc9b7 from 10 to 7 79s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Stopping container istio-proxy 79s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-d8glx 79s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-2msfs 78s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Stopping container istio-proxy 78s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Stopping container istio-proxy 76s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Readiness probe error: HTTP probe failed with statuscode: 503... 76s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Readiness probe error: HTTP probe failed with statuscode: 503... 76s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-d8glx Readiness probe failed: HTTP probe failed with statuscode: 503 76s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-9gb6c Readiness probe failed: HTTP probe failed with statuscode: 503 76s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Readiness probe error: HTTP probe failed with statuscode: 503... 76s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-2msfs Readiness probe failed: HTTP probe failed with statuscode: 503 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-26sm5 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-xvb52 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-7gt6k 63s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-9dccdc9b7 from 7 to 1 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-7gt6k Stopping container istio-proxy 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Stopping container istio-proxy 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-pfbqv Stopping container istio-proxy 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-xvb52 Stopping container istio-proxy 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-pfbqv 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-cf442 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Stopping container istio-proxy 63s Normal Killing pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Stopping container istio-proxy 63s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 1; reason: All metrics below target 63s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-9dccdc9b7 Deleted pod: maas-default-gateway-openshift-default-9dccdc9b7-lshsj 62s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Readiness probe error: HTTP probe failed with statuscode: 503... 62s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-26sm5 Readiness probe failed: HTTP probe failed with statuscode: 503 61s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Readiness probe error: HTTP probe failed with statuscode: 503... 61s Warning ProbeError pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Readiness probe error: HTTP probe failed with statuscode: 503... 61s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-cf442 Readiness probe failed: HTTP probe failed with statuscode: 503 61s Warning Unhealthy pod/maas-default-gateway-openshift-default-9dccdc9b7-lshsj Readiness probe failed: HTTP probe failed with statuscode: 503