=== events in openshift-ingress (2026-10-30T13:57:26Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 21m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 29m Normal Scheduled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Successfully assigned openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg to ip-10-0-142-194.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj to ip-10-0-141-232.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-fdnx5 to ip-10-0-143-38.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2 to ip-10-0-141-232.ec2.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6 to ip-10-0-141-232.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-m4kcl to ip-10-0-142-194.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-ndplc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc to ip-10-0-141-232.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-q8w96 to ip-10-0-142-194.ec2.internal 25m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 to ip-10-0-142-194.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr to ip-10-0-141-232.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Successfully assigned openshift-ingress/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s to ip-10-0-143-38.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Successfully assigned openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd to ip-10-0-142-194.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Successfully assigned openshift-ingress/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf to ip-10-0-143-38.ec2.internal 5m39s Normal Scheduled pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Successfully assigned openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 to ip-10-0-141-232.ec2.internal 9m7s Normal Scheduled pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Successfully assigned openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 to ip-10-0-141-232.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Successfully assigned openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq to ip-10-0-142-194.ec2.internal 32m Normal Scheduled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Successfully assigned openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv to ip-10-0-142-194.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tvm7w to ip-10-0-142-194.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x to ip-10-0-143-38.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-w27kj Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-w27kj to ip-10-0-143-38.ec2.internal 50m Normal Scheduled pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Successfully assigned openshift-ingress/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx to ip-10-0-143-38.ec2.internal 52m Normal Scheduled pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Successfully assigned openshift-ingress/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Successfully assigned openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Successfully assigned openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h to ip-10-0-143-38.ec2.internal 57m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh to ip-10-0-141-232.ec2.internal 49m Normal Scheduled pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Successfully assigned openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 to ip-10-0-141-232.ec2.internal 45m Normal Scheduled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Successfully assigned openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j to ip-10-0-142-194.ec2.internal 47m Normal Scheduled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Successfully assigned openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf to ip-10-0-141-232.ec2.internal 56m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb to ip-10-0-141-232.ec2.internal 49m Normal Scheduled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Successfully assigned openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj to ip-10-0-142-194.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Successfully assigned openshift-ingress/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz to ip-10-0-143-38.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs to ip-10-0-143-38.ec2.internal 46m Normal Scheduled pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Successfully assigned openshift-ingress/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l to ip-10-0-143-38.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 59m Normal Scheduled pod/payload-pre-processing-7b98bb678c-8lgfp Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-8lgfp to ip-10-0-143-38.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 52m Normal Scheduled pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft to ip-10-0-142-194.ec2.internal 63m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-ddspx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ddspx to ip-10-0-142-194.ec2.internal 25m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs to ip-10-0-141-232.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-9cf5j to ip-10-0-143-38.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Successfully assigned openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr to ip-10-0-142-194.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 48m Normal Scheduled pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Successfully assigned openshift-ingress/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl to ip-10-0-141-232.ec2.internal 72m Normal Scheduled pod/router-default-5db587c4bc-dttsw Successfully assigned openshift-ingress/router-default-5db587c4bc-dttsw to ip-10-0-142-194.ec2.internal 73m Warning FailedScheduling pod/router-default-5db587c4bc-dttsw no nodes available to schedule pods 79m Warning FailedScheduling pod/router-default-5db587c4bc-dttsw no nodes available to schedule pods 52m Normal Scheduled pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt to ip-10-0-143-38.ec2.internal 55m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Successfully assigned openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z to ip-10-0-143-38.ec2.internal 33m Normal Scheduled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Successfully assigned openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww to ip-10-0-143-38.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 26m Normal Scheduled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Successfully assigned openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Successfully assigned openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg to ip-10-0-143-38.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn to ip-10-0-141-232.ec2.internal 61m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-8982g Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g to ip-10-0-141-232.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Successfully assigned openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h to ip-10-0-142-194.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 to ip-10-0-141-232.ec2.internal 26m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd to ip-10-0-142-194.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt to ip-10-0-142-194.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 45m Normal Scheduled pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz to ip-10-0-141-232.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh to ip-10-0-142-194.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7mx6s to ip-10-0-143-38.ec2.internal 47m Normal Scheduled pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Successfully assigned openshift-ingress/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr to ip-10-0-143-38.ec2.internal 17m Normal Scheduled pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Successfully assigned openshift-ingress/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r to ip-10-0-143-38.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k to ip-10-0-143-38.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 to ip-10-0-142-194.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps to ip-10-0-143-38.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md to ip-10-0-142-194.ec2.internal 31m Normal Scheduled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Successfully assigned openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt to ip-10-0-143-38.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh to ip-10-0-143-38.ec2.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh to ip-10-0-143-38.ec2.internal 49m Normal Scheduled pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Successfully assigned openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw to ip-10-0-142-194.ec2.internal 44m Normal Scheduled pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Successfully assigned openshift-ingress/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf to ip-10-0-142-194.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Successfully assigned openshift-ingress/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg to ip-10-0-142-194.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj to ip-10-0-142-194.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v to ip-10-0-143-38.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq to ip-10-0-143-38.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 to ip-10-0-143-38.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw to ip-10-0-143-38.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j to ip-10-0-143-38.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw to ip-10-0-141-232.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq to ip-10-0-141-232.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r to ip-10-0-141-232.ec2.internal 44m Normal Scheduled pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Successfully assigned openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng to ip-10-0-141-232.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 to ip-10-0-143-38.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 to ip-10-0-141-232.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w to ip-10-0-143-38.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 to ip-10-0-143-38.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn to ip-10-0-143-38.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 to ip-10-0-143-38.ec2.internal 9m1s Normal Scheduled pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 to ip-10-0-141-232.ec2.internal 5m33s Normal Scheduled pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw to ip-10-0-141-232.ec2.internal 19m Normal Scheduled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Successfully assigned openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws to ip-10-0-142-194.ec2.internal 20m Warning FailedScheduling pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 57m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt to ip-10-0-141-232.ec2.internal 46m Normal Scheduled pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Successfully assigned openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg to ip-10-0-141-232.ec2.internal 23m Normal Scheduled pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Successfully assigned openshift-ingress/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Successfully assigned openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw to ip-10-0-142-194.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 to ip-10-0-141-232.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h to ip-10-0-142-194.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs to ip-10-0-142-194.ec2.internal 31m Normal Scheduled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Successfully assigned openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj to ip-10-0-141-232.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Successfully assigned openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn to ip-10-0-141-232.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Successfully assigned openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b to ip-10-0-142-194.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z to ip-10-0-141-232.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn to ip-10-0-141-232.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 to ip-10-0-141-232.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 to ip-10-0-143-38.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz to ip-10-0-142-194.ec2.internal 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p to ip-10-0-142-194.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 49m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb to ip-10-0-143-38.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss to ip-10-0-143-38.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 54m Normal Scheduled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Successfully assigned openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Successfully assigned openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk to ip-10-0-141-232.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz to ip-10-0-143-38.ec2.internal 50m Normal Scheduled pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Successfully assigned openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q to ip-10-0-143-38.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 to ip-10-0-142-194.ec2.internal 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n to ip-10-0-141-232.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 25m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 to ip-10-0-141-232.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp to ip-10-0-141-232.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 to ip-10-0-141-232.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p to ip-10-0-143-38.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 to ip-10-0-143-38.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk to ip-10-0-143-38.ec2.internal 51m Normal Scheduled pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Successfully assigned openshift-ingress/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 to ip-10-0-143-38.ec2.internal 23m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 to ip-10-0-141-232.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Successfully assigned openshift-ingress/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp to ip-10-0-141-232.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv to ip-10-0-141-232.ec2.internal 32m Normal Scheduled pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Successfully assigned openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 to ip-10-0-141-232.ec2.internal 51m Normal Scheduled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Successfully assigned openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn to ip-10-0-142-194.ec2.internal 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj to ip-10-0-141-232.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 to ip-10-0-142-194.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Successfully assigned openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 to ip-10-0-143-38.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k to ip-10-0-143-38.ec2.internal 47m Normal Scheduled pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Successfully assigned openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx to ip-10-0-142-194.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Successfully assigned openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv to ip-10-0-142-194.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l to ip-10-0-142-194.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 to ip-10-0-142-194.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb to ip-10-0-141-232.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 to ip-10-0-141-232.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n to ip-10-0-141-232.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Successfully assigned openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw to ip-10-0-141-232.ec2.internal 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm to ip-10-0-142-194.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 28m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 to ip-10-0-143-38.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h to ip-10-0-143-38.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb to ip-10-0-141-232.ec2.internal 25m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj to ip-10-0-141-232.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 to ip-10-0-142-194.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 to ip-10-0-142-194.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Successfully assigned openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng to ip-10-0-142-194.ec2.internal 19m Normal Scheduled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Successfully assigned openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z to ip-10-0-142-194.ec2.internal 19m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 49m Normal Scheduled pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Successfully assigned openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt to ip-10-0-141-232.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Successfully assigned openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 to ip-10-0-141-232.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Successfully assigned openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 to ip-10-0-141-232.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 38m Normal Scheduled pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Successfully assigned openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk to ip-10-0-141-232.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Successfully assigned openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl to ip-10-0-142-194.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w to ip-10-0-143-38.ec2.internal 20m Normal Scheduled pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Successfully assigned openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 to ip-10-0-141-232.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Successfully assigned openshift-ingress/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl to ip-10-0-143-38.ec2.internal 20m Warning FailedScheduling pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 32m Normal Scheduled pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Successfully assigned openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 to ip-10-0-143-38.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Normal Scheduled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Successfully assigned openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w to ip-10-0-141-232.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m to ip-10-0-141-232.ec2.internal 65m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-bvss5 Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-bvss5 to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j to ip-10-0-141-232.ec2.internal 17m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 to ip-10-0-141-232.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Successfully assigned openshift-ingress/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s to ip-10-0-143-38.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 16m Normal Scheduled pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 to ip-10-0-143-38.ec2.internal 17m Normal Scheduled pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 to ip-10-0-141-232.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 to ip-10-0-143-38.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 43m Normal Scheduled pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Successfully assigned openshift-ingress/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v to ip-10-0-143-38.ec2.internal 22m Warning FailedScheduling pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 25m Normal Scheduled pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp to ip-10-0-143-38.ec2.internal 17m Normal Scheduled pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 to ip-10-0-141-232.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z to ip-10-0-141-232.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p to ip-10-0-141-232.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 to ip-10-0-141-232.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb to ip-10-0-141-232.ec2.internal 25m Normal Scheduled pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df to ip-10-0-141-232.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl to ip-10-0-141-232.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv to ip-10-0-141-232.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd to ip-10-0-143-38.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg to ip-10-0-141-232.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj to ip-10-0-141-232.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d to ip-10-0-143-38.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz to ip-10-0-141-232.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh to ip-10-0-143-38.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk to ip-10-0-141-232.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 to ip-10-0-143-38.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m to ip-10-0-141-232.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb to ip-10-0-141-232.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql to ip-10-0-141-232.ec2.internal 39m Normal Scheduled pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Successfully assigned openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d to ip-10-0-143-38.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm to ip-10-0-141-232.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp to ip-10-0-143-38.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw to ip-10-0-141-232.ec2.internal 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck to ip-10-0-143-38.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Successfully assigned openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Successfully assigned openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g to ip-10-0-143-38.ec2.internal 23m Warning FailedScheduling pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Warning FailedScheduling pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 51m Normal Scheduled pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Successfully assigned openshift-ingress/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Successfully assigned openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q to ip-10-0-141-232.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Successfully assigned openshift-ingress/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc to ip-10-0-143-38.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Successfully assigned openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn to ip-10-0-141-232.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Successfully assigned openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk to ip-10-0-141-232.ec2.internal 16m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 to ip-10-0-141-232.ec2.internal 17m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 to ip-10-0-143-38.ec2.internal 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 40m Normal Scheduled pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Successfully assigned openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v to ip-10-0-141-232.ec2.internal 20m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 48m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26kf9 to ip-10-0-142-194.ec2.internal 20m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj to ip-10-0-141-232.ec2.internal 28m Normal Scheduled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Successfully assigned openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc to ip-10-0-141-232.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Successfully assigned openshift-ingress/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc to ip-10-0-143-38.ec2.internal 45m Normal Scheduled pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Successfully assigned openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl to ip-10-0-141-232.ec2.internal 46m Normal Scheduled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Successfully assigned openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Successfully assigned openshift-ingress/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-78h7p to ip-10-0-142-194.ec2.internal 34m Normal Scheduled pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Successfully assigned openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv to ip-10-0-143-38.ec2.internal 65m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dlbtt to ip-10-0-143-38.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r to ip-10-0-143-38.ec2.internal 37m Normal Scheduled pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Successfully assigned openshift-ingress/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k to ip-10-0-142-194.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj to ip-10-0-143-38.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Successfully assigned openshift-ingress/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc to ip-10-0-143-38.ec2.internal 61m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-7grvt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7grvt to ip-10-0-143-38.ec2.internal 28m Normal Scheduled pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Successfully assigned openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln to ip-10-0-141-232.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c to ip-10-0-141-232.ec2.internal 65m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 to ip-10-0-142-194.ec2.internal 40m Normal Scheduled pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Successfully assigned openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb to ip-10-0-141-232.ec2.internal 52m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f to ip-10-0-141-232.ec2.internal 55m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 to ip-10-0-142-194.ec2.internal 23m Normal Scheduled pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Successfully assigned openshift-ingress/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Successfully assigned openshift-ingress/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x to ip-10-0-143-38.ec2.internal 49m Normal Scheduled pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Successfully assigned openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg to ip-10-0-143-38.ec2.internal 31m Normal Scheduled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Successfully assigned openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t to ip-10-0-141-232.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Successfully assigned openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz to ip-10-0-141-232.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Successfully assigned openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s to ip-10-0-143-38.ec2.internal 42m Normal Scheduled pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Successfully assigned openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Successfully assigned openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 to ip-10-0-142-194.ec2.internal 49m Normal Scheduled pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Successfully assigned openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 to ip-10-0-141-232.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc to ip-10-0-141-232.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 to ip-10-0-141-232.ec2.internal 52m Normal Scheduled pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Successfully assigned openshift-ingress/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Successfully assigned openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 to ip-10-0-141-232.ec2.internal 28m Normal Scheduled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Successfully assigned openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz to ip-10-0-143-38.ec2.internal 37m Normal Scheduled pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Successfully assigned openshift-ingress/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Successfully assigned openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf to ip-10-0-143-38.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph to ip-10-0-141-232.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m to ip-10-0-143-38.ec2.internal 28m Normal Scheduled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Successfully assigned openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg to ip-10-0-142-194.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Successfully assigned openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m to ip-10-0-143-38.ec2.internal 59m Normal Scheduled pod/payload-processing-5f78687d5b-wrtq4 Successfully assigned openshift-ingress/payload-processing-5f78687d5b-wrtq4 to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Successfully assigned openshift-ingress/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b to ip-10-0-143-38.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-7462m Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7462m to ip-10-0-143-38.ec2.internal 33m Normal Scheduled pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Successfully assigned openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 to ip-10-0-141-232.ec2.internal 55m Normal Scheduled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Successfully assigned openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn to ip-10-0-143-38.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Successfully assigned openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv to ip-10-0-141-232.ec2.internal 55m Normal Scheduled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Successfully assigned openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn to ip-10-0-142-194.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Successfully assigned openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql to ip-10-0-141-232.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb to ip-10-0-142-194.ec2.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-52t9q Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-52t9q to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Successfully assigned openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 to ip-10-0-141-232.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl to ip-10-0-143-38.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 to ip-10-0-143-38.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 to ip-10-0-143-38.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4vfv9 to ip-10-0-142-194.ec2.internal 57m Normal Scheduled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Successfully assigned openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Successfully assigned openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx to ip-10-0-142-194.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Successfully assigned openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv to ip-10-0-142-194.ec2.internal 41m Normal Scheduled pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Successfully assigned openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj to ip-10-0-143-38.ec2.internal 28m Normal Scheduled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Successfully assigned openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj to ip-10-0-141-232.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 to ip-10-0-141-232.ec2.internal 32m Normal Scheduled pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Successfully assigned openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 to ip-10-0-141-232.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Successfully assigned openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl to ip-10-0-143-38.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch to ip-10-0-141-232.ec2.internal 65m Normal Scheduled pod/kube-auth-proxy-cbd4cb4f6-fl8vc Successfully assigned openshift-ingress/kube-auth-proxy-cbd4cb4f6-fl8vc to ip-10-0-143-38.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj to ip-10-0-143-38.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p to ip-10-0-141-232.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 to ip-10-0-143-38.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw to ip-10-0-141-232.ec2.internal 28m Normal Scheduled pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Successfully assigned openshift-ingress/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Successfully assigned openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g to ip-10-0-141-232.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Successfully assigned openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr to ip-10-0-141-232.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj to ip-10-0-141-232.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Successfully assigned openshift-ingress/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 to ip-10-0-142-194.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 to ip-10-0-143-38.ec2.internal 65m Normal Scheduled pod/kube-auth-proxy-cbd4cb4f6-7pqlr Successfully assigned openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr to ip-10-0-141-232.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Successfully assigned openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm to ip-10-0-143-38.ec2.internal 41m Normal Scheduled pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Successfully assigned openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj to ip-10-0-143-38.ec2.internal 56m Normal Scheduled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Successfully assigned openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 to ip-10-0-143-38.ec2.internal 5m33s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k to ip-10-0-141-232.ec2.internal 9m1s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh to ip-10-0-141-232.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 to ip-10-0-143-38.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j to ip-10-0-143-38.ec2.internal 43m Normal Scheduled pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Successfully assigned openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h to ip-10-0-141-232.ec2.internal 79m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-5db587c4bc from 0 to 1 79m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 79m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 79m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 79m Normal SuccessfulCreate replicaset/router-default-5db587c4bc Created pod: router-default-5db587c4bc-dttsw 72m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 72m Warning FailedMount pod/router-default-5db587c4bc-dttsw MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 72m Warning FailedMount pod/router-default-5db587c4bc-dttsw MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 72m Warning FailedMount pod/router-default-5db587c4bc-dttsw MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 72m Warning FailedMount pod/router-default-5db587c4bc-dttsw MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 72m Normal Started pod/router-default-5db587c4bc-dttsw Container started 72m Normal Created pod/router-default-5db587c4bc-dttsw Container created 72m Normal Pulled pod/router-default-5db587c4bc-dttsw Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 72m Normal AddedInterface pod/router-default-5db587c4bc-dttsw Add eth0 [10.132.0.16/23] from ovn-kubernetes 66m Normal SuccessfulCreate replicaset/kube-auth-proxy-cbd4cb4f6 Created pod: kube-auth-proxy-cbd4cb4f6-fl8vc 66m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-cbd4cb4f6 from 0 to 2 66m Normal SuccessfulCreate replicaset/kube-auth-proxy-cbd4cb4f6 Created pod: kube-auth-proxy-cbd4cb4f6-7pqlr 65m Normal Pulling pod/kube-auth-proxy-cbd4cb4f6-fl8vc Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 65m Normal AddedInterface pod/kube-auth-proxy-cbd4cb4f6-7pqlr Add eth0 [10.134.0.23/23] from ovn-kubernetes 65m Normal Pulling pod/kube-auth-proxy-cbd4cb4f6-7pqlr Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 65m Normal AddedInterface pod/kube-auth-proxy-cbd4cb4f6-fl8vc Add eth0 [10.133.0.25/23] from ovn-kubernetes 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 65m Normal Started pod/kube-auth-proxy-cbd4cb4f6-7pqlr Container started 65m Normal Pulled pod/kube-auth-proxy-cbd4cb4f6-fl8vc Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.455s (2.455s including waiting). Image size: 188904677 bytes. 65m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-bvss5 Add eth0 [10.133.0.26/23] from ovn-kubernetes 65m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-bvss5 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 65m Normal Pulled pod/kube-auth-proxy-cbd4cb4f6-7pqlr Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.346s (2.346s including waiting). Image size: 188904677 bytes. 65m Normal Created pod/kube-auth-proxy-cbd4cb4f6-7pqlr Container created 65m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-bvss5 65m Normal Created pod/kube-auth-proxy-cbd4cb4f6-fl8vc Container created 65m Normal Started pod/kube-auth-proxy-cbd4cb4f6-fl8vc Container started 65m Normal Created pod/istiod-openshift-gateway-7876bfcb74-bvss5 Container created 65m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-bvss5 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.508s (2.508s including waiting). Image size: 185476477 bytes. 65m Normal Started pod/istiod-openshift-gateway-7876bfcb74-bvss5 Container started 65m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-c574b7b6f from 0 to 1 65m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-bvss5 Readiness probe failed: HTTP probe failed with statuscode: 503 65m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-bvss5 Readiness probe error: HTTP probe failed with statuscode: 503... 65m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-c574b7b6f Created pod: data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 65m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Add eth0 [10.132.0.33/23] from ovn-kubernetes 65m Normal Pulling pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 65m Normal Started pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Container started 65m Normal Created pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Container created 65m Normal Pulled pod/data-science-gateway-data-science-gateway-class-c574b7b6f-fcbd8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.56s (2.56s including waiting). Image size: 183555333 bytes. 65m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Container created 65m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Add eth0 [10.133.0.27/23] from ovn-kubernetes 65m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 65m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-dlbtt 65m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Container started 65m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Readiness probe error: HTTP probe failed with statuscode: 503... 65m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-dlbtt Readiness probe failed: HTTP probe failed with statuscode: 503 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 65m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-bvss5 65m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-bvss5 Stopping container discovery 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 64m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 64m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 63m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 0 to 1 63m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 63m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-ddspx Add eth0 [10.132.0.44/23] from ovn-kubernetes 63m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-ddspx 62m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 62m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 61m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-7grvt 61m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-8982g 61m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 1 to 3 61m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-8982g Add eth0 [10.134.0.34/23] from ovn-kubernetes 61m Normal Pulling pod/maas-default-gateway-openshift-default-58c6585447-8982g Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 61m Normal Pulling pod/maas-default-gateway-openshift-default-58c6585447-7grvt Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 61m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-7grvt Add eth0 [10.133.0.37/23] from ovn-kubernetes 61m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-ddspx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 61m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-ddspx Container created 61m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-ddspx Container started 61m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-7grvt Container started 61m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-8982g Container created 61m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-7grvt Container created 61m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-8982g Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.736s (2.736s including waiting). Image size: 183555333 bytes. 61m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-8982g Container started 61m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-7grvt Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.512s (2.512s including waiting). Image size: 183555333 bytes. 61m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-ddspx Startup probe error: Get "http://10.132.0.44:15021/healthz/ready": dial tcp 10.132.0.44:15021: connect: connection refused... 61m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-ddspx Startup probe failed: Get "http://10.132.0.44:15021/healthz/ready": dial tcp 10.132.0.44:15021: connect: connection refused 61m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7grvt Startup probe failed: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused 61m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7grvt Startup probe error: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused... 61m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-8982g Startup probe error: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused... 61m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-8982g Startup probe failed: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused 60m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Container created 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Add eth0 [10.134.0.40/23] from ovn-kubernetes 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-q8w96 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Container started 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-qg8wr 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-7462m 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-7462m Add eth0 [10.133.0.41/23] from ovn-kubernetes 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-7462m Container started 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-7462m Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-7462m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Add eth0 [10.132.0.47/23] from ovn-kubernetes 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Container started 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Container created 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Startup probe error: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Startup probe failed: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused 59m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused... 59m Normal SuccessfulCreate replicaset/payload-processing-5f78687d5b Created pod: payload-processing-5f78687d5b-wrtq4 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7462m Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused... 59m Warning FailedCreate replicaset/payload-processing-5f78687d5b Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 59m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5f78687d5b from 0 to 1 59m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 59m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-8lgfp 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7462m Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused 59m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-8lgfp Add eth0 [10.133.0.43/23] from ovn-kubernetes 59m Normal Pulling pod/payload-pre-processing-7b98bb678c-8lgfp Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 59m Normal Pulling pod/payload-processing-5f78687d5b-wrtq4 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 59m Normal AddedInterface pod/payload-processing-5f78687d5b-wrtq4 Add eth0 [10.133.0.44/23] from ovn-kubernetes 59m Normal Started pod/payload-processing-5f78687d5b-wrtq4 Container started 59m Normal Pulled pod/payload-pre-processing-7b98bb678c-8lgfp Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.728s (2.728s including waiting). Image size: 125317101 bytes. 59m Normal Created pod/payload-processing-5f78687d5b-wrtq4 Container created 59m Normal Pulled pod/payload-processing-5f78687d5b-wrtq4 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.57s (2.57s including waiting). Image size: 125317101 bytes. 59m Normal Created pod/payload-pre-processing-7b98bb678c-8lgfp Container created 59m Normal Started pod/payload-pre-processing-7b98bb678c-8lgfp Container started 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7462m Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7462m Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-tvm7w 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Add eth0 [10.132.0.48/23] from ovn-kubernetes 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Add eth0 [10.134.0.41/23] from ovn-kubernetes 59m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 6 to 10 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-5tgdl 59m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Add eth0 [10.133.0.45/23] from ovn-kubernetes 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-w27kj Add eth0 [10.133.0.46/23] from ovn-kubernetes 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 (combined from similar events): Created pod: maas-default-gateway-openshift-default-58c6585447-w27kj 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-7mx6s 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-w27kj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Container started 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-w27kj Container started 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Container created 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Container started 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Container created 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-w27kj Container created 59m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Container started 59m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-w27kj Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused... 59m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-w27kj Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused... 58m Normal AddedLabel gateway/e2e-worker-w2-c2ce75 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-c2ce75 58m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled up replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 0 to 1 58m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled up replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 0 to 1 58m Normal AddedLabel gateway/e2e-worker-w3-09db16 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-09db16 58m Normal AddedLabel gateway/e2e-worker-w1-8b46c9 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-8b46c9 58m Normal SuccessfulCreate replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Created pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 58m Normal AddedLabel gateway/e2e-worker-w0-c90b9e Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-c90b9e 58m Normal AddedLabel gateway/e2e-worker-w4-67f56e Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-67f56e 58m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 58m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 0 to 1 58m Normal AddedLabel gateway/e2e-worker-w5-7a3603 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-7a3603 58m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz 58m Normal AddedLabel gateway/e2e-worker-w6-dd90cb Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-dd90cb 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-c2ce75-openshift-default-699569fc6c Created pod: e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g 58m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v 58m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 0 to 1 58m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 58m Normal AddedInterface pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Add eth0 [10.134.0.43/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 0 to 1 58m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled up replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 0 to 1 58m Normal AddedInterface pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Add eth0 [10.134.0.44/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 58m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Add eth0 [10.133.0.48/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/e2e-worker-w2-c2ce75-openshift-default Scaled up replica set e2e-worker-w2-c2ce75-openshift-default-699569fc6c from 0 to 1 58m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Add eth0 [10.133.0.47/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Add eth0 [10.133.0.50/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Add eth0 [10.134.0.45/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Add eth0 [10.133.0.49/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-09db16 Scaled up replica set payload-processing-e2e-worker-w3-09db16-7488884cbd from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-09db16-7488884cbd Created pod: payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-09db16-7488884cbd Error creating: pods "payload-processing-e2e-worker-w3-09db16-7488884cbd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-09db16: serviceaccount "payload-processing-e2e-worker-w3-09db16" not found 58m Normal Started pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Container started 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-09db16 Scaled up replica set payload-pre-processing-e2e-worker-w3-09db16-b446f45f9 from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9 Created pod: payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9 Error creating: pods "payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-09db16: serviceaccount "payload-processing-e2e-worker-w3-09db16" not found 58m Normal AddedInterface pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Add eth0 [10.134.0.46/23] from ovn-kubernetes 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Add eth0 [10.133.0.51/23] from ovn-kubernetes 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Container created 58m Normal Pulling pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-c2ce75 Scaled up replica set payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7 from 0 to 1 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-c2ce75 Scaled up replica set payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d Created pod: payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7 Created pod: payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7 Error creating: pods "payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-c2ce75: serviceaccount "payload-processing-e2e-worker-w2-c2ce75" not found 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d Error creating: pods "payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-c2ce75: serviceaccount "payload-processing-e2e-worker-w2-c2ce75" not found 58m Normal Started pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Container started 58m Normal Pulled pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-c90b9e Scaled up replica set payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff Created pod: payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff Error creating: pods "payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-c90b9e: serviceaccount "payload-processing-e2e-worker-w0-c90b9e" not found 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Started pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Container started 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Add eth0 [10.133.0.54/23] from ovn-kubernetes 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Add eth0 [10.133.0.52/23] from ovn-kubernetes 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54 Error creating: pods "payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-c90b9e: serviceaccount "payload-processing-e2e-worker-w0-c90b9e" not found 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54 Created pod: payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Started pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Container started 58m Normal Created pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Container created 58m Normal Pulled pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.11s (2.11s including waiting). Image size: 125317101 bytes. 58m Normal AddedInterface pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Add eth0 [10.133.0.53/23] from ovn-kubernetes 58m Normal Created pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Container created 58m Normal Created pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Container created 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-c90b9e Scaled up replica set payload-processing-e2e-worker-w0-c90b9e-5d546d5c54 from 0 to 1 58m Normal Pulled pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-dd90cb Scaled up replica set payload-processing-e2e-worker-w6-dd90cb-54889f9c66 from 0 to 1 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Add eth0 [10.132.0.49/23] from ovn-kubernetes 58m Normal Started pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Container started 58m Normal Created pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Container created 58m Normal Pulling pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-dd90cb-54889f9c66 Error creating: pods "payload-processing-e2e-worker-w6-dd90cb-54889f9c66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-dd90cb: serviceaccount "payload-processing-e2e-worker-w6-dd90cb" not found 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff Created pod: payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff Error creating: pods "payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-dd90cb: serviceaccount "payload-processing-e2e-worker-w6-dd90cb" not found 58m Normal Created pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Container started 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-dd90cb-54889f9c66 Created pod: payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-dd90cb Scaled up replica set payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff from 0 to 1 58m Normal AddedInterface pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Add eth0 [10.133.0.55/23] from ovn-kubernetes 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75 Error creating: pods "payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-7a3603: serviceaccount "payload-processing-e2e-worker-w5-7a3603" not found 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd Error creating: pods "payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-7a3603: serviceaccount "payload-processing-e2e-worker-w5-7a3603" not found 58m Normal Started pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Container started 58m Normal Created pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Container created 58m Normal Pulled pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Add eth0 [10.133.0.56/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-7a3603 Scaled up replica set payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd from 0 to 1 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-7a3603 Scaled up replica set payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75 from 0 to 1 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-67f56e Scaled up replica set payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd Created pod: payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Add eth0 [10.133.0.57/23] from ovn-kubernetes 58m Normal Pulling pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Normal AddedInterface pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Add eth0 [10.132.0.50/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-67f56e Scaled up replica set payload-processing-e2e-worker-w4-67f56e-56998884fb from 0 to 1 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Container started 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75 Created pod: payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-67f56e-56998884fb Created pod: payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 58m Normal AddedInterface pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Add eth0 [10.132.0.51/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c Created pod: payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj 58m Normal Started pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Container started 58m Normal Created pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Container started 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-67f56e-56998884fb Error creating: pods "payload-processing-e2e-worker-w4-67f56e-56998884fb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-67f56e: serviceaccount "payload-processing-e2e-worker-w4-67f56e" not found 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Add eth0 [10.133.0.58/23] from ovn-kubernetes 58m Normal Started pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Container started 58m Normal Pulled pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 302ms (302ms including waiting). Image size: 125317101 bytes. 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Container created 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c Error creating: pods "payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-67f56e: serviceaccount "payload-processing-e2e-worker-w4-67f56e" not found 58m Normal Created pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Container started 58m Normal Created pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Container created 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.042s (2.042s including waiting). Image size: 125317101 bytes. 58m Normal Pulled pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5 Error creating: pods "payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-8b46c9: serviceaccount "payload-processing-e2e-worker-w1-8b46c9" not found 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-8b46c9 Scaled up replica set payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5 from 0 to 1 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6 Error creating: pods "payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-8b46c9: serviceaccount "payload-processing-e2e-worker-w1-8b46c9" not found 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6 Created pod: payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5 Created pod: payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-8b46c9 Scaled up replica set payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6 from 0 to 1 58m Normal Pulled pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Add eth0 [10.133.0.59/23] from ovn-kubernetes 58m Normal Created pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Container created 58m Normal Started pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Container started 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Add eth0 [10.133.0.60/23] from ovn-kubernetes 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Container started 58m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Container created 58m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Container started 58m Normal Started pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Container started 58m Normal Pulled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Container created 58m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Container started 58m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Container started 58m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Container created 58m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Container created 58m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 58m Normal Created pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Container created 58m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Readiness probe failed: Get "http://10.134.0.44:15021/healthz/ready": read tcp 10.134.0.2:56048->10.134.0.44:15021: read: connection reset by peer 58m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Readiness probe error: Get "http://10.134.0.44:15021/healthz/ready": read tcp 10.134.0.2:56048->10.134.0.44:15021: read: connection reset by peer... 58m Normal Started pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Container started 58m Normal Pulled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused 58m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Startup probe failed: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Startup probe error: Get "http://10.133.0.48:15021/healthz/ready": dial tcp 10.133.0.48:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Startup probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused 58m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Startup probe error: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused... 58m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Container started 58m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Container created 58m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Container created 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Container started 58m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 58m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 58m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 58m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 57m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Startup probe error: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused... 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled up replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 1 to 2 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh 57m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Container created 57m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Add eth0 [10.134.0.54/23] from ovn-kubernetes 57m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Container started 57m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Startup probe error: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Startup probe failed: Get "http://10.134.0.54:15021/healthz/ready": dial tcp 10.134.0.54:15021: connect: connection refused 57m Normal SuccessfulCreate replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Created pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled up replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 1 to 2 57m Normal AddedInterface pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Add eth0 [10.133.0.70/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Container created 57m Normal Started pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Container started 57m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled up replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 1 to 2 57m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Add eth0 [10.134.0.55/23] from ovn-kubernetes 57m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Container created 57m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Container started 57m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Startup probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Startup probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Startup probe error: Get "http://10.134.0.55:15021/healthz/ready": dial tcp 10.134.0.55:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Startup probe failed: Get "http://10.134.0.55:15021/healthz/ready": dial tcp 10.134.0.55:15021: connect: connection refused 56m Normal SuccessfulCreate replicaset/e2e-worker-w2-c2ce75-openshift-default-699569fc6c Created pod: e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w2-c2ce75-openshift-default Scaled up replica set e2e-worker-w2-c2ce75-openshift-default-699569fc6c from 1 to 2 56m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled up replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 2 to 4 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h 56m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Container started 56m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Container created 56m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Add eth0 [10.134.0.56/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Container started 56m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Container started 56m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Container created 56m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Add eth0 [10.132.0.67/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Add eth0 [10.133.0.72/23] from ovn-kubernetes 56m Normal Created pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Container created 56m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused... 56m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Startup probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Startup probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 56m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 1 to 2 56m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 1 to 2 56m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb 56m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Container started 56m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Container created 56m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Add eth0 [10.134.0.57/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Add eth0 [10.133.0.75/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Container started 56m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Container created 56m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Startup probe failed: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Startup probe error: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused... 56m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled up replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 2 to 4 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 56m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Container started 56m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Add eth0 [10.132.0.71/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Container created 56m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Add eth0 [10.134.0.58/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Container started 56m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Container created 56m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused... 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl 56m Normal SuccessfulCreate replicaset/e2e-worker-w2-c2ce75-openshift-default-699569fc6c Created pod: e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 1 to 2 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w2-c2ce75-openshift-default Scaled up replica set e2e-worker-w2-c2ce75-openshift-default-699569fc6c from 2 to 3 56m Normal Created pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Container created 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md 56m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Container started 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled up replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 4 to 6 56m Normal AddedInterface pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Add eth0 [10.132.0.74/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 56m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Container created 56m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Add eth0 [10.134.0.59/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Container started 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g 56m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Add eth0 [10.132.0.75/23] from ovn-kubernetes 56m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Container started 56m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Container started 56m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Container created 56m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Container created 56m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Add eth0 [10.133.0.82/23] from ovn-kubernetes 55m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 55m Normal Created pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Container created 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal Started pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Container started 55m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled up replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 2 to 3 55m Normal AddedInterface pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Add eth0 [10.132.0.76/23] from ovn-kubernetes 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Created pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn 55m Normal Pulled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Container started 55m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Container created 55m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Container started 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 2 to 3 55m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Add eth0 [10.132.0.77/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Add eth0 [10.133.0.86/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Container created 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 2 to 4 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn 55m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Container started 55m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Container created 55m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Add eth0 [10.132.0.78/23] from ovn-kubernetes 55m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Startup probe error: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Startup probe failed: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 2 to 4 55m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Container created 55m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Container started 55m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Container started 55m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Container created 55m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Add eth0 [10.133.0.88/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Add eth0 [10.132.0.80/23] from ovn-kubernetes 55m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Startup probe error: Get "http://10.133.0.88:15021/healthz/ready": dial tcp 10.133.0.88:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Startup probe failed: Get "http://10.133.0.88:15021/healthz/ready": dial tcp 10.133.0.88:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 55m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Add eth0 [10.134.0.63/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n 55m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 4 to 5 55m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Container started 55m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Container created 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Startup probe error: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Startup probe failed: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Startup probe error: Get "http://10.134.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Startup probe failed: Get "http://10.134.0.63:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Container created 54m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Container created 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Add eth0 [10.134.0.66/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Add eth0 [10.132.0.81/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Add eth0 [10.133.0.92/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Add eth0 [10.134.0.65/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 54m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Add eth0 [10.133.0.93/23] from ovn-kubernetes 54m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 5 to 8 54m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Container started 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz 54m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Container started 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Add eth0 [10.132.0.82/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 3 to 6 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Container created 54m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Container started 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Container created 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Container started 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Container started 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Container started 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Container created 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Container created 54m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Startup probe error: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Startup probe failed: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Startup probe failed: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Startup probe error: Get "http://10.133.0.93:15021/healthz/ready": dial tcp 10.133.0.93:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Startup probe error: Get "http://10.133.0.92:15021/healthz/ready": dial tcp 10.133.0.92:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Startup probe failed: Get "http://10.133.0.92:15021/healthz/ready": dial tcp 10.133.0.92:15021: connect: connection refused 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 4 to 5 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 6 to 7 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 7; reason: All metrics below target 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled up replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 6 to 10 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Add eth0 [10.133.0.96/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Container created 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Add eth0 [10.134.0.67/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf (combined from similar events): Created pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Container started 54m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Container created 54m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Container created 54m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Container started 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Container started 54m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Add eth0 [10.132.0.83/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Container started 54m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Add eth0 [10.133.0.99/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Container created 54m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Container created 54m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Container started 54m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Add eth0 [10.134.0.68/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Add eth0 [10.133.0.98/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Container created 54m Normal Started pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Container started 54m Normal Pulled pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Startup probe failed: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Startup probe error: Get "http://10.134.0.68:15021/healthz/ready": dial tcp 10.134.0.68:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Startup probe failed: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Startup probe error: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Startup probe failed: Get "http://10.133.0.98:15021/healthz/ready": dial tcp 10.133.0.98:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Startup probe failed: Get "http://10.133.0.99:15021/healthz/ready": dial tcp 10.133.0.99:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Startup probe error: Get "http://10.133.0.96:15021/healthz/ready": dial tcp 10.133.0.96:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Startup probe error: Get "http://10.133.0.99:15021/healthz/ready": dial tcp 10.133.0.99:15021: connect: connection refused... 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 5 to 7 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 54m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled up replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 8 to 10 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 (combined from similar events): Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Created pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Add eth0 [10.132.0.86/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Container started 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Container created 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Readiness probe failed: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Container created 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Add eth0 [10.133.0.102/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Add eth0 [10.134.0.69/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Container started 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Readiness probe error: Get "http://10.133.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Normal AddedInterface pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Add eth0 [10.132.0.87/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Container created 54m Normal Started pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Container started 54m Normal Pulled pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Container created 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Container started 54m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Startup probe error: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Startup probe error: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Startup probe failed: Get "http://10.134.0.69:15021/healthz/ready": dial tcp 10.134.0.69:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Startup probe failed: Get "http://10.132.0.87:15021/healthz/ready": dial tcp 10.132.0.87:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Startup probe error: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Startup probe failed: Get "http://10.133.0.102:15021/healthz/ready": dial tcp 10.133.0.102:15021: connect: connection refused 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled up replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 7 to 10 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 (combined from similar events): Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Created pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Add eth0 [10.132.0.88/23] from ovn-kubernetes 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled up replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 7 to 10 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Container started 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Container created 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Container started 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Add eth0 [10.134.0.70/23] from ovn-kubernetes 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 (combined from similar events): Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Container created 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Created pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Container created 54m Normal AddedInterface pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Add eth0 [10.133.0.103/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Pulled pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Container started 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Container created 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Container started 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Add eth0 [10.132.0.89/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Add eth0 [10.134.0.71/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Container started 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Container created 54m Normal Pulled pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Add eth0 [10.133.0.105/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Container started 53m Normal Created pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Container created 53m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Startup probe error: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Startup probe error: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Startup probe failed: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Startup probe failed: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Startup probe error: Get "http://10.133.0.105:15021/healthz/ready": dial tcp 10.133.0.105:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Startup probe failed: Get "http://10.134.0.70:15021/healthz/ready": dial tcp 10.134.0.70:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Readiness probe error: Get "http://10.132.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Readiness probe failed: Get "http://10.132.0.87:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Readiness probe error: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Readiness probe failed: Get "http://10.132.0.89:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Readiness probe error: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Readiness probe failed: Get "http://10.134.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 53m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Readiness probe error: Get "http://10.133.0.102:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Readiness probe failed: Get "http://10.133.0.102:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-q8n9f 52m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 52m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 52m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Add eth0 [10.134.0.72/23] from ovn-kubernetes 52m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 52m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Container created 52m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Container started 52m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.596s (2.596s including waiting). Image size: 185476477 bytes. 52m Normal AddedLabel gateway/e2e-ait-68573cef Added label istio.io/rev=openshift-gateway to gateway e2e-ait-68573cef 52m Normal ScalingReplicaSet deployment/e2e-ait-68573cef-openshift-default Scaled up replica set e2e-ait-68573cef-openshift-default-549b6dd947 from 0 to 1 52m Normal SuccessfulCreate replicaset/e2e-ait-68573cef-openshift-default-549b6dd947 Created pod: e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s 52m Normal AddedInterface pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Add eth0 [10.133.0.120/23] from ovn-kubernetes 52m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Readiness probe error: Get "http://10.132.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Readiness probe error: Get "http://10.134.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Readiness probe failed: Get "http://10.132.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Readiness probe failed: Get "http://10.134.0.55:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Readiness probe failed: Get "http://10.134.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Readiness probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Readiness probe failed: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Readiness probe error: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Readiness probe error: Get "http://10.132.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Readiness probe failed: Get "http://10.132.0.78:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Readiness probe failed: Get "http://10.132.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Readiness probe error: Get "http://10.132.0.75:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Readiness probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Readiness probe error: Get "http://10.134.0.58:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Readiness probe failed: Get "http://10.133.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Readiness probe error: Get "http://10.133.0.82:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Readiness probe error: Get "http://10.134.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Readiness probe failed: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Readiness probe error: Get "http://10.134.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Readiness probe failed: Get "http://10.134.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Readiness probe error: Get "http://10.133.0.105:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Readiness probe failed: Get "http://10.133.0.105:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87 Created pod: payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb 52m Normal AddedLabel gateway/e2e-ait-752d3a58 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-752d3a58 52m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87 Error creating: pods "payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-68573cef: serviceaccount "payload-processing-e2e-ait-68573cef" not found 52m Normal SuccessfulCreate replicaset/e2e-ait-752d3a58-openshift-default-fb496cb47 Created pod: e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz 52m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-68573cef Scaled up replica set payload-processing-e2e-ait-68573cef-864d98dbf6 from 0 to 1 52m Normal ScalingReplicaSet deployment/e2e-ait-752d3a58-openshift-default Scaled up replica set e2e-ait-752d3a58-openshift-default-fb496cb47 from 0 to 1 52m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-68573cef Scaled up replica set payload-pre-processing-e2e-ait-68573cef-5b99d4bf87 from 0 to 1 52m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-68573cef-864d98dbf6 Created pod: payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn 52m Warning FailedCreate replicaset/payload-processing-e2e-ait-68573cef-864d98dbf6 Error creating: pods "payload-processing-e2e-ait-68573cef-864d98dbf6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-68573cef: serviceaccount "payload-processing-e2e-ait-68573cef" not found 52m Normal AddedInterface pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Add eth0 [10.133.0.123/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Add eth0 [10.133.0.121/23] from ovn-kubernetes 52m Normal Started pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Container started 52m Normal Created pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Container created 52m Normal Pulled pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal Started pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Container started 52m Normal Created pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Container created 52m Normal Pulled pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Add eth0 [10.133.0.124/23] from ovn-kubernetes 52m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-752d3a58 Scaled up replica set payload-pre-processing-e2e-ait-752d3a58-77b49c547 from 0 to 1 52m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-752d3a58 Scaled up replica set payload-processing-e2e-ait-752d3a58-568ccdcfc6 from 0 to 1 52m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-752d3a58-77b49c547 Created pod: payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p 52m Warning FailedCreate replicaset/payload-processing-e2e-ait-752d3a58-568ccdcfc6 Error creating: pods "payload-processing-e2e-ait-752d3a58-568ccdcfc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-752d3a58: serviceaccount "payload-processing-e2e-ait-752d3a58" not found 52m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-752d3a58-77b49c547 Error creating: pods "payload-pre-processing-e2e-ait-752d3a58-77b49c547-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-752d3a58: serviceaccount "payload-processing-e2e-ait-752d3a58" not found 52m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-752d3a58-568ccdcfc6 Created pod: payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x 52m Normal AddedInterface pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Add eth0 [10.133.0.127/23] from ovn-kubernetes 52m Normal Started pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Container started 52m Normal AddedInterface pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Add eth0 [10.133.0.126/23] from ovn-kubernetes 52m Normal Started pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Container started 52m Normal Created pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Container created 52m Normal Pulled pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal Pulled pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 52m Normal Created pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Container created 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Created pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv 52m Normal Pulled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt 52m Normal AddedInterface pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Add eth0 [10.133.0.131/23] from ovn-kubernetes 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j 52m Normal Started pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Container started 52m Normal Created pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Container created 52m Normal Created pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Container created 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Created pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf 52m Normal Pulled pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled up replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 3 to 5 52m Normal AddedInterface pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Add eth0 [10.132.0.100/23] from ovn-kubernetes 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 52m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled up replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 4 to 8 52m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Container started 52m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Add eth0 [10.132.0.101/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Add eth0 [10.134.0.73/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Container created 52m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Container started 52m Normal Started pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Container started 52m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Add eth0 [10.132.0.102/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Container created 52m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Container started 52m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Container created 52m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Container started 52m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Container created 52m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Add eth0 [10.133.0.132/23] from ovn-kubernetes 51m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Readiness probe failed: Get "http://10.133.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Readiness probe failed: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Readiness probe error: Get "http://10.134.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Readiness probe error: Get "http://10.133.0.92:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Readiness probe error: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-8982g Readiness probe error: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-8982g Readiness probe failed: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Readiness probe failed: Get "http://10.132.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Startup probe failed: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Startup probe error: Get "http://10.132.0.100:15021/healthz/ready": dial tcp 10.132.0.100:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Startup probe failed: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused 51m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Readiness probe failed: timeout: health rpc did not complete within 1s 51m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Readiness probe failed: timeout: failed to connect service "10.133.0.57:50052" within 1s: context deadline exceeded 51m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Readiness probe error: Get "http://10.133.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Readiness probe failed: Get "http://10.133.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Startup probe failed: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Startup probe error: Get "http://10.132.0.102:15021/healthz/ready": dial tcp 10.132.0.102:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Startup probe error: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Startup probe error: Get "http://10.133.0.131:15021/healthz/ready": dial tcp 10.133.0.131:15021: connect: connection refused... 51m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7462m Readiness probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Startup probe failed: Get "http://10.133.0.131:15021/healthz/ready": dial tcp 10.133.0.131:15021: connect: connection refused 51m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7462m Readiness probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Startup probe error: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Startup probe failed: Get "http://10.133.0.132:15021/healthz/ready": dial tcp 10.133.0.132:15021: connect: connection refused 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Startup probe failed: Get "http://10.133.0.132:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Startup probe error: Get "http://10.133.0.132:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp 51m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 10; reason: All metrics below target 51m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled up replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 8 to 10 51m Normal SuccessfulCreate replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 (combined from similar events): Created pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv 51m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Container created 51m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Add eth0 [10.134.0.74/23] from ovn-kubernetes 51m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Container started 51m Normal AddedInterface pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Add eth0 [10.133.0.133/23] from ovn-kubernetes 51m Normal Created pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Container created 51m Normal Pulled pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Started pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Container started 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 51m Normal Created pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Container created 51m Normal Started pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Container started 51m Normal Pulled pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Startup probe error: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Readiness probe failed: Get "http://10.133.0.120:15021/healthz/ready": read tcp 10.133.0.2:55170->10.133.0.120:15021: read: connection reset by peer 51m Normal Pulled pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Container created 51m Normal Started pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Container started 51m Warning ProbeError pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Readiness probe error: Get "http://10.133.0.120:15021/healthz/ready": read tcp 10.133.0.2:55170->10.133.0.120:15021: read: connection reset by peer... 51m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Startup probe failed: Get "http://10.133.0.133:15021/healthz/ready": dial tcp 10.133.0.133:15021: connect: connection refused 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-752d3a58-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-752d3a58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning Unhealthy pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Startup probe failed: Get "http://10.133.0.120:15021/healthz/ready": dial tcp 10.133.0.120:15021: connect: connection refused 51m Warning ProbeError pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Startup probe error: Get "http://10.133.0.120:15021/healthz/ready": dial tcp 10.133.0.120:15021: connect: connection refused... 51m Warning ProbeError pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Startup probe error: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Startup probe failed: Get "http://10.133.0.121:15021/healthz/ready": dial tcp 10.133.0.121:15021: connect: connection refused 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-68573cef-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-68573cef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Normal Killing pod/payload-pre-processing-e2e-ait-68573cef-5b99d4bf87-v8sxb Stopping container payload-pre-processing 51m Normal Killing pod/payload-processing-e2e-ait-68573cef-864d98dbf6-7k9zn Stopping container payload-processing 51m Normal Killing pod/payload-pre-processing-e2e-ait-752d3a58-77b49c547-whg7p Stopping container payload-pre-processing 51m Normal Killing pod/payload-processing-e2e-ait-752d3a58-568ccdcfc6-nwm4x Stopping container payload-processing 51m Normal Killing pod/e2e-ait-68573cef-openshift-default-549b6dd947-tkn8s Stopping container istio-proxy 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-752d3a58-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-752d3a58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Normal SuccessfulCreate replicaset/e2e-derive-5af2de67-openshift-default-59979bb889 Created pod: e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 51m Normal Killing pod/e2e-ait-752d3a58-openshift-default-fb496cb47-ngsjz Stopping container istio-proxy 51m Normal AddedLabel gateway/e2e-derive-5af2de67 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-5af2de67 51m Normal ScalingReplicaSet deployment/e2e-derive-5af2de67-openshift-default Scaled up replica set e2e-derive-5af2de67-openshift-default-59979bb889 from 0 to 1 51m Normal AddedInterface pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Add eth0 [10.133.0.134/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-5af2de67 Scaled up replica set payload-processing-e2e-derive-5af2de67-dd66d7776 from 0 to 1 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-5af2de67-5bc4df475 Error creating: pods "payload-pre-processing-e2e-derive-5af2de67-5bc4df475-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-5af2de67: serviceaccount "payload-processing-e2e-derive-5af2de67" not found 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-5af2de67-dd66d7776 Created pod: payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 51m Warning FailedCreate replicaset/payload-processing-e2e-derive-5af2de67-dd66d7776 Error creating: pods "payload-processing-e2e-derive-5af2de67-dd66d7776-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-5af2de67: serviceaccount "payload-processing-e2e-derive-5af2de67" not found 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-5af2de67-5bc4df475 Created pod: payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-5af2de67 Scaled up replica set payload-pre-processing-e2e-derive-5af2de67-5bc4df475 from 0 to 1 51m Normal Created pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Container created 51m Normal Pulled pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Add eth0 [10.133.0.136/23] from ovn-kubernetes 51m Normal AddedInterface pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Add eth0 [10.133.0.135/23] from ovn-kubernetes 51m Normal Started pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Container started 51m Normal Pulled pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Container created 51m Normal Started pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Container started 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-5af2de67-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-5af2de67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Normal SuccessfulCreate replicaset/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477 Created pod: e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx 50m Normal AddedLabel gateway/e2e-migrate-85be4bb9-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-85be4bb9-gw 50m Normal ScalingReplicaSet deployment/e2e-migrate-85be4bb9-gw-openshift-default Scaled up replica set e2e-migrate-85be4bb9-gw-openshift-default-5744b87477 from 0 to 1 50m Normal AddedInterface pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Add eth0 [10.133.0.138/23] from ovn-kubernetes 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-85be4bb9 Scaled up replica set payload-processing-e2e-migrate-85be4bb9-f76b758b6 from 0 to 1 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-85be4bb9 Scaled up replica set payload-pre-processing-e2e-migrate-85be4bb9-68445ff947 from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947 Created pod: payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx 50m Warning FailedCreate replicaset/payload-processing-e2e-migrate-85be4bb9-f76b758b6 Error creating: pods "payload-processing-e2e-migrate-85be4bb9-f76b758b6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-85be4bb9: serviceaccount "payload-processing-e2e-migrate-85be4bb9" not found 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947 Error creating: pods "payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-85be4bb9: serviceaccount "payload-processing-e2e-migrate-85be4bb9" not found 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-85be4bb9-f76b758b6 Created pod: payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt 50m Normal AddedInterface pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Add eth0 [10.133.0.140/23] from ovn-kubernetes 50m Normal Started pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Container started 50m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Add eth0 [10.133.0.139/23] from ovn-kubernetes 50m Normal Pulled pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Container created 50m Normal Pulled pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Started pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Container started 50m Normal Created pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Container created 50m Normal Created pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Container created 50m Normal Pulled pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Started pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Container started 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-5af2de67-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-5af2de67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning Unhealthy pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Startup probe failed: Get "http://10.133.0.134:15021/healthz/ready": dial tcp 10.133.0.134:15021: connect: connection refused 50m Warning ProbeError pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Startup probe error: Get "http://10.133.0.134:15021/healthz/ready": dial tcp 10.133.0.134:15021: connect: connection refused... 50m Normal Killing pod/payload-processing-e2e-derive-5af2de67-dd66d7776-c4dm6 Stopping container payload-processing 50m Normal Killing pod/payload-pre-processing-e2e-derive-5af2de67-5bc4df475-h8kkh Stopping container payload-pre-processing 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-ddspx Readiness probe failed: Get "http://10.132.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-ddspx Readiness probe error: Get "http://10.132.0.44:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Readiness probe error: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Readiness probe failed: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Readiness probe failed: Get "http://10.134.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-w27kj Readiness probe failed: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-w27kj Readiness probe error: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Readiness probe error: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Readiness probe error: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Readiness probe failed: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal Killing pod/e2e-derive-5af2de67-openshift-default-59979bb889-9xsm9 Stopping container istio-proxy 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-85be4bb9-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-85be4bb9-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 Created pod: e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q 50m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-2afe58-openshift-default Scaled up replica set e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 from 0 to 1 50m Normal AddedLabel gateway/e2e-shared-a-w1-2afe58 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-2afe58 50m Normal AddedInterface pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Add eth0 [10.133.0.142/23] from ovn-kubernetes 50m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876 Error creating: pods "payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58: serviceaccount "payload-processing-e2e-shared-a-w1-2afe58" not found 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-2afe58 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599 from 0 to 1 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-2afe58 Scaled up replica set payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876 from 0 to 1 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-2afe58: serviceaccount "payload-processing-e2e-shared-a-w1-2afe58" not found 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876 Created pod: payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h 50m Normal Pulled pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Add eth0 [10.133.0.143/23] from ovn-kubernetes 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599 Created pod: payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m 50m Normal Created pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Container created 50m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Add eth0 [10.133.0.144/23] from ovn-kubernetes 50m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Container created 50m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Container started 50m Normal Started pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Container started 49m Normal Created pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Container created 49m Normal Pulled pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Started pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Container started 49m Normal SuccessfulCreate replicaset/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 Created pod: e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d 49m Normal ScalingReplicaSet deployment/e2e-mt-1c0e6c42-openshift-default Scaled up replica set e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 from 0 to 1 49m Normal AddedLabel gateway/e2e-mt-1c0e6c42 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-1c0e6c42 49m Warning Unhealthy pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Startup probe failed: Get "http://10.133.0.138:15021/healthz/ready": dial tcp 10.133.0.138:15021: connect: connection refused 49m Warning ProbeError pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Startup probe error: Get "http://10.133.0.138:15021/healthz/ready": dial tcp 10.133.0.138:15021: connect: connection refused... 49m Normal AddedInterface pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Add eth0 [10.133.0.145/23] from ovn-kubernetes 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-85be4bb9-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 Error creating: pods "payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1c0e6c42: serviceaccount "payload-processing-e2e-mt-1c0e6c42" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 Created pod: payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-1c0e6c42 Scaled up replica set payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 from 0 to 1 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-85be4bb9-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedCreate replicaset/payload-processing-e2e-mt-1c0e6c42-dd9d66d84 Error creating: pods "payload-processing-e2e-mt-1c0e6c42-dd9d66d84-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1c0e6c42: serviceaccount "payload-processing-e2e-mt-1c0e6c42" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-1c0e6c42-dd9d66d84 Created pod: payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-1c0e6c42 Scaled up replica set payload-processing-e2e-mt-1c0e6c42-dd9d66d84 from 0 to 1 49m Normal Started pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Container started 49m Normal Started pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Container started 49m Normal Created pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Container created 49m Normal Pulled pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Add eth0 [10.134.0.76/23] from ovn-kubernetes 49m Normal Killing pod/payload-pre-processing-e2e-migrate-85be4bb9-68445ff947-vtxfx Stopping container payload-pre-processing 49m Normal Created pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Container created 49m Normal Killing pod/payload-processing-e2e-migrate-85be4bb9-f76b758b6-d4hvt Stopping container payload-processing 49m Normal AddedInterface pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Add eth0 [10.134.0.75/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Killing pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Stopping container istio-proxy 49m Normal AddedLabel gateway/e2e-shared-a-w5-bcd76c Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-bcd76c 49m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-bcd76c-openshift-default Scaled up replica set e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc from 0 to 1 49m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc Created pod: e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs 49m Normal AddedInterface pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Add eth0 [10.133.0.146/23] from ovn-kubernetes 49m Warning Unhealthy pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Readiness probe failed: HTTP probe failed with statuscode: 503 49m Warning ProbeError pod/e2e-migrate-85be4bb9-gw-openshift-default-5744b87477-bkjpx Readiness probe error: HTTP probe failed with statuscode: 503... 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-bcd76c Scaled up replica set payload-processing-e2e-shared-a-w5-bcd76c-7b78578448 from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-bcd76c Scaled up replica set payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655 from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655 Created pod: payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655 Error creating: pods "payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c: serviceaccount "payload-processing-e2e-shared-a-w5-bcd76c" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448 Created pod: payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448 Error creating: pods "payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-bcd76c: serviceaccount "payload-processing-e2e-shared-a-w5-bcd76c" not found 49m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Add eth0 [10.133.0.147/23] from ovn-kubernetes 49m Normal Pulled pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Add eth0 [10.133.0.148/23] from ovn-kubernetes 49m Normal Started pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Container started 49m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Container started 49m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Container created 49m Normal Created pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Container created 49m Warning Unhealthy pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Readiness probe failed: Get "http://10.133.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 49m Warning ProbeError pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Readiness probe error: Get "http://10.133.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 49m Normal AddedLabel gateway/e2e-shared-b-w1-7a71ca Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-7a71ca 49m Normal Pulled pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Container created 49m Normal Started pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Container started 49m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc Created pod: e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 49m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-7a71ca-openshift-default Scaled up replica set e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc from 0 to 1 49m Normal AddedInterface pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Add eth0 [10.134.0.77/23] from ovn-kubernetes 49m Warning ProbeError pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Startup probe error: Get "http://10.133.0.142:15021/healthz/ready": dial tcp 10.133.0.142:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Startup probe failed: Get "http://10.133.0.142:15021/healthz/ready": dial tcp 10.133.0.142:15021: connect: connection refused 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-7a71ca Scaled up replica set payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48 from 0 to 1 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c Error creating: pods "payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca: serviceaccount "payload-processing-e2e-shared-b-w1-7a71ca" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c Created pod: payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch 49m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48 Created pod: payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48 Error creating: pods "payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-7a71ca: serviceaccount "payload-processing-e2e-shared-b-w1-7a71ca" not found 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Add eth0 [10.134.0.78/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-7a71ca Scaled up replica set payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c from 0 to 1 49m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Container created 49m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Add eth0 [10.133.0.151/23] from ovn-kubernetes 49m Normal Started pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Container started 49m Normal Created pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Container created 49m Normal Pulled pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Container started 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Normal AddedLabel gateway/e2e-mt-17fc433e Added label istio.io/rev=openshift-gateway to gateway e2e-mt-17fc433e 49m Normal ScalingReplicaSet deployment/e2e-mt-17fc433e-openshift-default Scaled up replica set e2e-mt-17fc433e-openshift-default-79d8c9969 from 0 to 1 49m Normal SuccessfulCreate replicaset/e2e-mt-17fc433e-openshift-default-79d8c9969 Created pod: e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 49m Normal AddedInterface pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Add eth0 [10.134.0.79/23] from ovn-kubernetes 49m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 Created pod: e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg 49m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-0b2c92-openshift-default Scaled up replica set e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 from 0 to 1 49m Normal AddedLabel gateway/e2e-shared-a-w6-0b2c92 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-0b2c92 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-17fc433e-58fcb84879 Error creating: pods "payload-pre-processing-e2e-mt-17fc433e-58fcb84879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-17fc433e: serviceaccount "payload-processing-e2e-mt-17fc433e" not found 49m Normal AddedInterface pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Add eth0 [10.133.0.152/23] from ovn-kubernetes 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-17fc433e Scaled up replica set payload-pre-processing-e2e-mt-17fc433e-58fcb84879 from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-17fc433e Scaled up replica set payload-processing-e2e-mt-17fc433e-78585b7c6d from 0 to 1 48m Warning FailedCreate replicaset/payload-processing-e2e-mt-17fc433e-78585b7c6d Error creating: pods "payload-processing-e2e-mt-17fc433e-78585b7c6d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-17fc433e: serviceaccount "payload-processing-e2e-mt-17fc433e" not found 48m Normal Pulled pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-17fc433e-58fcb84879 Created pod: payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-17fc433e-78585b7c6d Created pod: payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl 48m Normal AddedInterface pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Add eth0 [10.134.0.80/23] from ovn-kubernetes 48m Normal Pulled pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Container started 48m Normal Started pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Container started 48m Normal Created pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Container created 48m Normal AddedInterface pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Add eth0 [10.133.0.153/23] from ovn-kubernetes 48m Normal Created pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Container created 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-0b2c92 Scaled up replica set payload-processing-e2e-shared-a-w6-0b2c92-b489c67b from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87 Created pod: payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87 Error creating: pods "payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92: serviceaccount "payload-processing-e2e-shared-a-w6-0b2c92" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b Created pod: payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n 48m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b Error creating: pods "payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92: serviceaccount "payload-processing-e2e-shared-a-w6-0b2c92" not found 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-0b2c92 Scaled up replica set payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87 from 0 to 1 48m Normal Pulled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Container started 48m Normal Created pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Container created 48m Normal Created pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Container created 48m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Add eth0 [10.134.0.82/23] from ovn-kubernetes 48m Normal Started pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Container started 48m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Add eth0 [10.134.0.81/23] from ovn-kubernetes 48m Normal Started pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Container started 48m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Container created 48m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Pulled pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Startup probe error: Get "http://10.133.0.145:15021/healthz/ready": dial tcp 10.133.0.145:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Startup probe failed: Get "http://10.133.0.145:15021/healthz/ready": dial tcp 10.133.0.145:15021: connect: connection refused 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 48m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-26kf9 48m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Add eth0 [10.132.0.113/23] from ovn-kubernetes 48m Normal AddedLabel gateway/e2e-shared-b-w5-0e874e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-0e874e 48m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0e874e-openshift-default Scaled up replica set e2e-shared-b-w5-0e874e-openshift-default-594f895d85 from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Created pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal AddedInterface pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Add eth0 [10.133.0.154/23] from ovn-kubernetes 48m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.267s (3.267s including waiting). Image size: 185476477 bytes. 48m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Container created 48m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Container started 48m Warning Unhealthy pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Readiness probe failed: Get "http://10.133.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Warning ProbeError pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Readiness probe error: Get "http://10.133.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Pulled pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Container started 48m Normal Created pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Container created 48m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b Error creating: pods "payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e: serviceaccount "payload-processing-e2e-shared-b-w5-0e874e" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b Created pod: payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9 Error creating: pods "payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e: serviceaccount "payload-processing-e2e-shared-b-w5-0e874e" not found 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-0e874e Scaled up replica set payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-0e874e Scaled up replica set payload-processing-e2e-shared-b-w5-0e874e-f9f47688b from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9 Created pod: payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw 48m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Add eth0 [10.134.0.85/23] from ovn-kubernetes 48m Normal Pulled pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Container started 48m Normal Created pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Container created 48m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Container started 48m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Add eth0 [10.134.0.84/23] from ovn-kubernetes 48m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Container created 48m Warning ProbeError pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Startup probe error: Get "http://10.133.0.146:15021/healthz/ready": dial tcp 10.133.0.146:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Startup probe failed: Get "http://10.133.0.146:15021/healthz/ready": dial tcp 10.133.0.146:15021: connect: connection refused 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal AddedLabel gateway/e2e-api-a-75abce Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-75abce 48m Normal ScalingReplicaSet deployment/e2e-api-a-75abce-openshift-default Scaled up replica set e2e-api-a-75abce-openshift-default-749fdc447f from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-api-a-75abce-openshift-default-749fdc447f Created pod: e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r 48m Normal AddedInterface pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Add eth0 [10.133.0.156/23] from ovn-kubernetes 48m Normal Started pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Container started 48m Normal Created pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Container created 48m Normal Pulled pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Started pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Container started 48m Normal Pulled pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Created pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Container created 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-75abce Scaled up replica set payload-pre-processing-e2e-api-a-75abce-dfb9f7f55 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-75abce Scaled up replica set payload-processing-e2e-api-a-75abce-bd9687d56 from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55 Created pod: payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55 Error creating: pods "payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-75abce: serviceaccount "payload-processing-e2e-api-a-75abce" not found 48m Warning FailedCreate replicaset/payload-processing-e2e-api-a-75abce-bd9687d56 Error creating: pods "payload-processing-e2e-api-a-75abce-bd9687d56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-75abce: serviceaccount "payload-processing-e2e-api-a-75abce" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-75abce-bd9687d56 Created pod: payload-processing-e2e-api-a-75abce-bd9687d56-scc2b 48m Normal Pulled pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 48m Normal Created pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Container created 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal Started pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Container started 48m Normal Created pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Container created 48m Warning ProbeError pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 48m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Add eth0 [10.134.0.87/23] from ovn-kubernetes 48m Normal Started pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Container started 48m Normal AddedInterface pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Add eth0 [10.133.0.157/23] from ovn-kubernetes 48m Warning Unhealthy pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 48m Normal Pulled pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning Unhealthy pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 47m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7 Created pod: e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx 47m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-acd125-openshift-default Scaled up replica set e2e-shared-a-w2-acd125-openshift-default-688b6f7db7 from 0 to 1 47m Normal AddedLabel gateway/e2e-shared-a-w2-acd125 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-acd125 47m Normal AddedInterface pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Add eth0 [10.132.0.114/23] from ovn-kubernetes 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-acd125-565744995b Created pod: payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5 Created pod: payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5 Error creating: pods "payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-acd125: serviceaccount "payload-processing-e2e-shared-a-w2-acd125" not found 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-acd125 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5 from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-acd125-565744995b Error creating: pods "payload-processing-e2e-shared-a-w2-acd125-565744995b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-acd125: serviceaccount "payload-processing-e2e-shared-a-w2-acd125" not found 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-acd125 Scaled up replica set payload-processing-e2e-shared-a-w2-acd125-565744995b from 0 to 1 47m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Add eth0 [10.133.0.159/23] from ovn-kubernetes 47m Normal Pulled pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Add eth0 [10.133.0.158/23] from ovn-kubernetes 47m Normal Started pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Container started 47m Normal Created pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Container created 47m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Container created 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Container started 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-17fc433e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-17fc433e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf Created pod: e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd 47m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-3e8ee4-openshift-default Scaled up replica set e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf from 0 to 1 47m Normal AddedLabel gateway/e2e-shared-b-w6-3e8ee4 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-3e8ee4 47m Normal AddedInterface pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Add eth0 [10.132.0.115/23] from ovn-kubernetes 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 47m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Stopping container istio-proxy 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 5; reason: All metrics below target 47m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg 47m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 47m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 47m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 Stopping container istio-proxy 47m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-sbvm7 Stopping container istio-proxy 47m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl Stopping container istio-proxy 47m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled down replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 10 to 5 47m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-zzqcg Stopping container istio-proxy 47m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-r2rxl 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-3e8ee4 Scaled up replica set payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59 from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98 Error creating: pods "payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4: serviceaccount "payload-processing-e2e-shared-b-w6-3e8ee4" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98 Created pod: payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-3e8ee4 Scaled up replica set payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98 from 0 to 1 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59 Error creating: pods "payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4: serviceaccount "payload-processing-e2e-shared-b-w6-3e8ee4" not found 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59 Created pod: payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj 47m Normal Created pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Container created 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Add eth0 [10.134.0.89/23] from ovn-kubernetes 47m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Container started 47m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Container created 47m Normal Started pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Container started 47m Normal Pulled pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Add eth0 [10.134.0.90/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Readiness probe failed: Get "http://10.133.0.47:15021/healthz/ready": read tcp 10.133.0.2:48504->10.133.0.47:15021: read: connection reset by peer 47m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-bh455 Readiness probe error: Get "http://10.133.0.47:15021/healthz/ready": read tcp 10.133.0.2:48504->10.133.0.47:15021: read: connection reset by peer... 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-17fc433e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-17fc433e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal Started pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Container started 47m Normal Pulled pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Container created 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-acd125-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-acd125-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning ProbeError pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Startup probe error: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Startup probe failed: Get "http://10.133.0.152:15021/healthz/ready": dial tcp 10.133.0.152:15021: connect: connection refused 47m Normal Killing pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Stopping container payload-processing 47m Normal Killing pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Stopping container payload-pre-processing 47m Normal AddedLabel gateway/e2e-api-a-f3160a Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-f3160a 47m Normal ScalingReplicaSet deployment/e2e-mt-1c0e6c42-openshift-default Scaled up replica set e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 from 1 to 2 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 47m Normal SuccessfulCreate replicaset/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 Created pod: e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf 47m Normal ScalingReplicaSet deployment/e2e-api-a-f3160a-openshift-default Scaled up replica set e2e-api-a-f3160a-openshift-default-67c76678c from 0 to 1 47m Normal Created pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Container created 47m Normal Started pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Container started 47m Normal SuccessfulCreate replicaset/e2e-api-a-f3160a-openshift-default-67c76678c Created pod: e2e-api-a-f3160a-openshift-default-67c76678c-77hqr 47m Normal AddedInterface pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Add eth0 [10.134.0.91/23] from ovn-kubernetes 47m Normal Pulled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Add eth0 [10.133.0.163/23] from ovn-kubernetes 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md 47m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 47m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kb7md Stopping container istio-proxy 47m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq Stopping container istio-proxy 47m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-vp2gq 47m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g 47m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-hbf4g Stopping container istio-proxy 47m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 47m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2l9k4 Stopping container istio-proxy 47m Warning FailedCreate replicaset/payload-processing-e2e-api-a-f3160a-6bc4cf46b4 Error creating: pods "payload-processing-e2e-api-a-f3160a-6bc4cf46b4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f3160a: serviceaccount "payload-processing-e2e-api-a-f3160a" not found 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-f3160a-6bc4cf46b4 Created pod: payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9 Error creating: pods "payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-f3160a: serviceaccount "payload-processing-e2e-api-a-f3160a" not found 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9 Created pod: payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-f3160a Scaled up replica set payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-f3160a Scaled up replica set payload-processing-e2e-api-a-f3160a-6bc4cf46b4 from 0 to 1 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 5; reason: All metrics below target 47m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled down replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 10 to 5 47m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-2mzk8 Stopping container istio-proxy 47m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Add eth0 [10.133.0.166/23] from ovn-kubernetes 47m Warning Unhealthy pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Startup probe failed: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused 47m Warning ProbeError pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Startup probe error: Get "http://10.134.0.91:15021/healthz/ready": dial tcp 10.134.0.91:15021: connect: connection refused... 47m Normal Pulled pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Container created 47m Normal Started pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Container started 47m Normal Created pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Container created 47m Normal AddedInterface pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Add eth0 [10.133.0.167/23] from ovn-kubernetes 47m Normal Started pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Container started 47m Normal Pulled pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Container created 47m Normal Pulled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Started pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Container started 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning Unhealthy pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Startup probe failed: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Startup probe error: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused... 47m Warning Unhealthy pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 47m Warning Unhealthy pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 47m Normal SuccessfulDelete replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Deleted pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf 47m Normal Started pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Container started 47m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled down replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 5 to 4 47m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 4; reason: All metrics below target 47m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-qpdvf Stopping container istio-proxy 47m Normal Killing pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-nk5cf Stopping container istio-proxy 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 8; reason: All metrics below target 47m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled down replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 10 to 8 47m Normal Pulled pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Created pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Container created 47m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph 46m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-bmqph Stopping container istio-proxy 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-75abce-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-75abce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning ProbeError pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Startup probe error: Get "http://10.133.0.156:15021/healthz/ready": dial tcp 10.133.0.156:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Startup probe failed: Get "http://10.133.0.156:15021/healthz/ready": dial tcp 10.133.0.156:15021: connect: connection refused 46m Normal AddedLabel gateway/e2e-api-b-31e10b Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-31e10b 46m Normal ScalingReplicaSet deployment/e2e-api-b-31e10b-openshift-default Scaled up replica set e2e-api-b-31e10b-openshift-default-5c5846b755 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-api-b-31e10b-openshift-default-5c5846b755 Created pod: e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz 46m Normal AddedInterface pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Add eth0 [10.133.0.169/23] from ovn-kubernetes 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 6; reason: All metrics below target 46m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled down replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 8 to 6 46m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn 46m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m 46m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-2xdqn Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m Stopping container istio-proxy 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-31e10b Scaled up replica set payload-processing-e2e-api-b-31e10b-59678cd5fb from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-31e10b-59678cd5fb Created pod: payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc 46m Warning FailedCreate replicaset/payload-processing-e2e-api-b-31e10b-59678cd5fb Error creating: pods "payload-processing-e2e-api-b-31e10b-59678cd5fb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-31e10b: serviceaccount "payload-processing-e2e-api-b-31e10b" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-31e10b-77669586dc Error creating: pods "payload-pre-processing-e2e-api-b-31e10b-77669586dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-31e10b: serviceaccount "payload-processing-e2e-api-b-31e10b" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-31e10b-77669586dc Created pod: payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-31e10b Scaled up replica set payload-pre-processing-e2e-api-b-31e10b-77669586dc from 0 to 1 46m Warning ProbeError pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Readiness probe error: Get "http://10.132.0.114:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 46m Normal Created pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Container created 46m Warning Unhealthy pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Readiness probe failed: Get "http://10.132.0.114:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Normal Started pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Container started 46m Normal Created pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Container created 46m Normal Pulled pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Add eth0 [10.133.0.171/23] from ovn-kubernetes 46m Normal Started pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Container started 46m Normal Pulled pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Add eth0 [10.133.0.170/23] from ovn-kubernetes 46m Normal Pulled pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Container created 46m Normal Pulled pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Container created 46m Normal Started pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Container started 46m Normal Started pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Container started 46m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Created pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj 46m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled up replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 0 to 1 46m Normal AddedLabel gateway/e2e-shared-b-w2-e9d641 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-e9d641 46m Normal AddedInterface pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Add eth0 [10.133.0.173/23] from ovn-kubernetes 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning Unhealthy pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Startup probe failed: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused 46m Warning ProbeError pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Startup probe error: Get "http://10.132.0.114:15021/healthz/ready": dial tcp 10.132.0.114:15021: connect: connection refused... 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75 Error creating: pods "payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641: serviceaccount "payload-processing-e2e-shared-b-w2-e9d641" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75 Created pod: payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-e9d641 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75 from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb Error creating: pods "payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-e9d641: serviceaccount "payload-processing-e2e-shared-b-w2-e9d641" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb Created pod: payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-e9d641 Scaled up replica set payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb from 0 to 1 46m Warning Unhealthy pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Startup probe failed: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused 46m Warning ProbeError pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Startup probe error: Get "http://10.132.0.115:15021/healthz/ready": dial tcp 10.132.0.115:15021: connect: connection refused... 46m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Container created 46m Normal Started pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Container started 46m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Container started 46m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Add eth0 [10.133.0.174/23] from ovn-kubernetes 46m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Add eth0 [10.133.0.175/23] from ovn-kubernetes 46m Normal Pulled pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal SuccessfulDelete replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Deleted pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 3; reason: All metrics below target 46m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled down replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 4 to 3 46m Normal Created pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Container created 46m Normal Killing pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-ljrjn Stopping container istio-proxy 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-75abce-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-75abce-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal AddedLabel gateway/e2e-api-a-5b9e8e Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-5b9e8e 46m Normal ScalingReplicaSet deployment/e2e-api-a-5b9e8e-openshift-default Scaled up replica set e2e-api-a-5b9e8e-openshift-default-77bfd57d69 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-api-a-5b9e8e-openshift-default-77bfd57d69 Created pod: e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l 46m Normal AddedInterface pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Add eth0 [10.133.0.176/23] from ovn-kubernetes 46m Normal Killing pod/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c Stopping container payload-pre-processing 46m Normal Killing pod/payload-processing-e2e-mt-17fc433e-78585b7c6d-swkrl Stopping container payload-processing 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-acd125-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-acd125-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-5b9e8e Scaled up replica set payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4 from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4 Error creating: pods "payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-5b9e8e: serviceaccount "payload-processing-e2e-api-a-5b9e8e" not found 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5 Created pod: payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5 Error creating: pods "payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-5b9e8e: serviceaccount "payload-processing-e2e-api-a-5b9e8e" not found 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-5b9e8e Scaled up replica set payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4 Created pod: payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc 46m Normal Created pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Container created 46m Normal Pulled pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Add eth0 [10.133.0.178/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Container started 46m Normal Pulled pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Container created 46m Normal Started pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Container started 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Add eth0 [10.133.0.177/23] from ovn-kubernetes 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-5b9e8e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-5b9e8e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal Pulled pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Container started 46m Normal Created pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Container created 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-3e8ee4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-3e8ee4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 46m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp Stopping container istio-proxy 46m Normal Killing pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 Stopping container istio-proxy 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 5; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb 46m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled down replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 10 to 5 46m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n 46m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-p2qlz Stopping container istio-proxy 45m Warning Unhealthy pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Startup probe failed: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused 45m Warning ProbeError pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Startup probe error: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused... 45m Normal AddedLabel gateway/e2e-api-a-abbc6c Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-abbc6c 45m Normal ScalingReplicaSet deployment/e2e-api-a-abbc6c-openshift-default Scaled up replica set e2e-api-a-abbc6c-openshift-default-9f4f4d7ff from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff Created pod: e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw 45m Normal AddedInterface pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Add eth0 [10.134.0.92/23] from ovn-kubernetes 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-1c0e6c42 Scaled up replica set payload-processing-e2e-mt-1c0e6c42-dd9d66d84 from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-1c0e6c42 Scaled up replica set payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-mt-1c0e6c42-dd9d66d84 Error creating: pods "payload-processing-e2e-mt-1c0e6c42-dd9d66d84-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1c0e6c42: serviceaccount "payload-processing-e2e-mt-1c0e6c42" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-1c0e6c42-dd9d66d84 Created pod: payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 Error creating: pods "payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-1c0e6c42: serviceaccount "payload-processing-e2e-mt-1c0e6c42" not found 45m Normal AddedInterface pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Add eth0 [10.134.0.93/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479 Created pod: payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 45m Normal Pulled pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Container created 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-abbc6c Scaled up replica set payload-processing-e2e-api-a-abbc6c-64bbd756cb from 0 to 1 45m Normal Created pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Container created 45m Normal Pulled pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Container started 45m Normal AddedInterface pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Add eth0 [10.134.0.94/23] from ovn-kubernetes 45m Normal Started pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Container started 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f3160a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-abbc6c Scaled up replica set payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f from 0 to 1 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-1c0e6c42-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 45m Normal SuccessfulCreate replicaset/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 Created pod: e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f Created pod: payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f Error creating: pods "payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-abbc6c: serviceaccount "payload-processing-e2e-api-a-abbc6c" not found 45m Normal ScalingReplicaSet deployment/e2e-mt-1c0e6c42-openshift-default Scaled up replica set e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65 from 2 to 3 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f3160a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedCreate replicaset/payload-processing-e2e-api-a-abbc6c-64bbd756cb Error creating: pods "payload-processing-e2e-api-a-abbc6c-64bbd756cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-abbc6c: serviceaccount "payload-processing-e2e-api-a-abbc6c" not found 45m Normal Pulled pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Add eth0 [10.132.0.123/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-abbc6c-64bbd756cb Created pod: payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl 45m Normal Started pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Container started 45m Normal Created pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Container created 45m Normal Started pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Container started 45m Normal Created pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Container created 45m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Add eth0 [10.134.0.95/23] from ovn-kubernetes 45m Normal Pulled pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Pulled pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Container created 45m Normal Started pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Container started 45m Normal AddedInterface pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Add eth0 [10.134.0.96/23] from ovn-kubernetes 45m Normal AddedLabel gateway/e2e-api-b-716b4c Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-716b4c 45m Normal ScalingReplicaSet deployment/e2e-api-b-716b4c-openshift-default Scaled up replica set e2e-api-b-716b4c-openshift-default-55bb8ffb77 from 0 to 1 45m Normal SuccessfulCreate replicaset/e2e-api-b-716b4c-openshift-default-55bb8ffb77 Created pod: e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk 45m Normal AddedInterface pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Add eth0 [10.134.0.98/23] from ovn-kubernetes 45m Warning ProbeError pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Startup probe error: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Startup probe failed: Get "http://10.132.0.123:15021/healthz/ready": dial tcp 10.132.0.123:15021: connect: connection refused 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-716b4c Scaled up replica set payload-pre-processing-e2e-api-b-716b4c-7df4b898c from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-716b4c Scaled up replica set payload-processing-e2e-api-b-716b4c-85c6b65cb6 from 0 to 1 45m Warning FailedCreate replicaset/payload-processing-e2e-api-b-716b4c-85c6b65cb6 Error creating: pods "payload-processing-e2e-api-b-716b4c-85c6b65cb6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-716b4c: serviceaccount "payload-processing-e2e-api-b-716b4c" not found 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-716b4c-7df4b898c Created pod: payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-716b4c-7df4b898c Error creating: pods "payload-pre-processing-e2e-api-b-716b4c-7df4b898c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-716b4c: serviceaccount "payload-processing-e2e-api-b-716b4c" not found 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-716b4c-85c6b65cb6 Created pod: payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q 45m Normal Pulled pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Created pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Container created 45m Normal Started pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Container started 45m Normal Pulled pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Add eth0 [10.133.0.179/23] from ovn-kubernetes 45m Normal Created pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Container created 45m Normal AddedInterface pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Add eth0 [10.134.0.99/23] from ovn-kubernetes 45m Normal Started pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Container started 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-f3160a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-f3160a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal SuccessfulCreate replicaset/e2e-api-b-31e10b-openshift-default-5c5846b755 Created pod: e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn 45m Normal ScalingReplicaSet deployment/e2e-api-b-31e10b-openshift-default Scaled up replica set e2e-api-b-31e10b-openshift-default-5c5846b755 from 1 to 2 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-api-b-31e10b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 45m Normal AddedInterface pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Add eth0 [10.134.0.100/23] from ovn-kubernetes 45m Normal Created pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Container created 45m Normal Pulled pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Container started 45m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv 45m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp 45m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt 45m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled down replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 10 to 7 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 7; reason: All metrics below target 45m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv Stopping container istio-proxy 45m Warning Unhealthy pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Readiness probe failed: Get "http://10.133.0.169:15021/healthz/ready": read tcp 10.133.0.2:56186->10.133.0.169:15021: read: connection reset by peer 45m Warning ProbeError pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Readiness probe error: Get "http://10.133.0.169:15021/healthz/ready": read tcp 10.133.0.2:56186->10.133.0.169:15021: read: connection reset by peer... 45m Normal Started pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Container started 45m Normal Created pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Container created 45m Normal Pulled pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Warning ProbeError pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Startup probe error: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Startup probe failed: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused 45m Warning ProbeError pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Startup probe error: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Startup probe failed: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused 45m Normal Pulled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Started pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Container started 45m Normal Created pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Container created 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal Killing pod/payload-processing-e2e-api-a-75abce-bd9687d56-scc2b Stopping container payload-processing 45m Normal Killing pod/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn Stopping container payload-pre-processing 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-31e10b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-31e10b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Startup probe error: Get "http://10.133.0.173:15021/healthz/ready": dial tcp 10.133.0.173:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Startup probe failed: Get "http://10.133.0.173:15021/healthz/ready": dial tcp 10.133.0.173:15021: connect: connection refused 44m Normal AddedLabel gateway/e2e-inf-a-2a3e01 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-2a3e01 44m Normal SuccessfulCreate replicaset/e2e-inf-a-2a3e01-openshift-default-56dfc4f47 Created pod: e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 44m Normal ScalingReplicaSet deployment/e2e-inf-a-2a3e01-openshift-default Scaled up replica set e2e-inf-a-2a3e01-openshift-default-56dfc4f47 from 0 to 1 44m Normal AddedInterface pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Add eth0 [10.133.0.181/23] from ovn-kubernetes 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-2a3e01 Scaled up replica set payload-processing-e2e-inf-a-2a3e01-8b78d7988 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf Created pod: payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq 44m Normal Started pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Container started 44m Normal Pulled pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Container created 44m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-2a3e01-8b78d7988 Error creating: pods "payload-processing-e2e-inf-a-2a3e01-8b78d7988-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2a3e01: serviceaccount "payload-processing-e2e-inf-a-2a3e01" not found 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf Error creating: pods "payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2a3e01: serviceaccount "payload-processing-e2e-inf-a-2a3e01" not found 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-2a3e01-8b78d7988 Created pod: payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-2a3e01 Scaled up replica set payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf from 0 to 1 44m Normal Started pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Container started 44m Normal AddedInterface pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Add eth0 [10.134.0.104/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Container created 44m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Add eth0 [10.133.0.182/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Container created 44m Normal Started pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Container started 44m Warning Unhealthy pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Startup probe failed: Get "http://10.133.0.176:15021/healthz/ready": dial tcp 10.133.0.176:15021: connect: connection refused 44m Warning ProbeError pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Startup probe error: Get "http://10.133.0.176:15021/healthz/ready": dial tcp 10.133.0.176:15021: connect: connection refused... 44m Normal ScalingReplicaSet deployment/e2e-api-b-14e8a4-openshift-default Scaled up replica set e2e-api-b-14e8a4-openshift-default-5d6dcf965 from 0 to 1 44m Normal AddedLabel gateway/e2e-api-b-14e8a4 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-14e8a4 44m Normal SuccessfulCreate replicaset/e2e-api-b-14e8a4-openshift-default-5d6dcf965 Created pod: e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng 44m Normal AddedInterface pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Add eth0 [10.134.0.105/23] from ovn-kubernetes 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-14e8a4 Scaled up replica set payload-processing-e2e-api-b-14e8a4-7fb9b9d5d from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d Created pod: payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-14e8a4 Scaled up replica set payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f Created pod: payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f Error creating: pods "payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-14e8a4: serviceaccount "payload-processing-e2e-api-b-14e8a4" not found 44m Warning FailedCreate replicaset/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d Error creating: pods "payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-14e8a4: serviceaccount "payload-processing-e2e-api-b-14e8a4" not found 44m Normal Created pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Container created 44m Normal Started pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Container started 44m Normal AddedInterface pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Add eth0 [10.134.0.106/23] from ovn-kubernetes 44m Normal Pulled pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Container started 44m Normal Created pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Container created 44m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Add eth0 [10.133.0.183/23] from ovn-kubernetes 44m Normal Pulled pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Warning ProbeError pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Readiness probe error: Get "http://10.134.0.92:15021/healthz/ready": read tcp 10.134.0.2:49690->10.134.0.92:15021: read: connection reset by peer... 44m Warning Unhealthy pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Readiness probe failed: Get "http://10.134.0.92:15021/healthz/ready": read tcp 10.134.0.2:49690->10.134.0.92:15021: read: connection reset by peer 44m Normal Started pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Container started 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal Created pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Container created 44m Normal Started pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Container started 44m Normal Pulled pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal Created pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Container created 44m Normal Pulled pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-5b9e8e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-5b9e8e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-716b4c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-716b4c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning ProbeError pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Startup probe error: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Startup probe failed: Get "http://10.134.0.98:15021/healthz/ready": dial tcp 10.134.0.98:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Startup probe failed: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused 44m Warning ProbeError pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Startup probe error: Get "http://10.134.0.98:15021/healthz/ready": dial tcp 10.134.0.98:15021: connect: connection refused... 44m Normal ScalingReplicaSet deployment/e2e-api-b-11cfa6-openshift-default Scaled up replica set e2e-api-b-11cfa6-openshift-default-6dd9b4cb7 from 0 to 1 44m Normal SuccessfulCreate replicaset/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7 Created pod: e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf 44m Normal AddedLabel gateway/e2e-api-b-11cfa6 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-11cfa6 44m Normal AddedInterface pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Add eth0 [10.132.0.128/23] from ovn-kubernetes 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-abbc6c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-abbc6c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75 Error creating: pods "payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-11cfa6: serviceaccount "payload-processing-e2e-api-b-11cfa6" not found 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-11cfa6 Scaled up replica set payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75 from 0 to 1 44m Warning FailedCreate replicaset/payload-processing-e2e-api-b-11cfa6-689984f4b8 Error creating: pods "payload-processing-e2e-api-b-11cfa6-689984f4b8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-11cfa6: serviceaccount "payload-processing-e2e-api-b-11cfa6" not found 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-11cfa6 Scaled up replica set payload-processing-e2e-api-b-11cfa6-689984f4b8 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75 Created pod: payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps 44m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Add eth0 [10.133.0.184/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-11cfa6-689984f4b8 Created pod: payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk 44m Normal Started pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Container started 44m Normal Pulled pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Pulled pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Container created 44m Normal Started pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Container started 44m Normal AddedInterface pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Add eth0 [10.134.0.107/23] from ovn-kubernetes 44m Normal Created pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Container created 44m Normal Killing pod/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9 Stopping container payload-pre-processing 44m Normal Killing pod/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5 Stopping container payload-processing 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-716b4c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-716b4c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-abbc6c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-abbc6c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-processing-e2e-api-a-f3160a-6bc4cf46b4-p49zc Stopping container payload-processing 43m Normal Killing pod/payload-pre-processing-e2e-api-a-f3160a-ffb45f7c9-mxvhc Stopping container payload-pre-processing 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-11cfa6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-11cfa6-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 5; reason: All metrics below target 43m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default Scaled down replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 6 to 5 43m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb Stopping container istio-proxy 43m Normal AddedLabel gateway/e2e-inf-a-4a8ee1 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-4a8ee1 43m Normal ScalingReplicaSet deployment/e2e-inf-a-4a8ee1-openshift-default Scaled up replica set e2e-inf-a-4a8ee1-openshift-default-566547cf95 from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-inf-a-4a8ee1-openshift-default-566547cf95 Created pod: e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v 43m Normal AddedInterface pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Add eth0 [10.133.0.187/23] from ovn-kubernetes 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7 Error creating: pods "payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-4a8ee1: serviceaccount "payload-processing-e2e-inf-a-4a8ee1" not found 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6 Created pod: payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm 43m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6 Error creating: pods "payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-4a8ee1: serviceaccount "payload-processing-e2e-inf-a-4a8ee1" not found 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-4a8ee1 Scaled up replica set payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6 from 0 to 1 43m Normal Killing pod/e2e-api-a-75abce-openshift-default-749fdc447f-l9w9r Stopping container istio-proxy 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-4a8ee1 Scaled up replica set payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7 from 0 to 1 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7 Created pod: payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr 43m Normal AddedInterface pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Add eth0 [10.134.0.109/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Container started 43m Normal Created pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Container created 43m Normal Created pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Container created 43m Normal Pulled pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal Pulled pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Add eth0 [10.134.0.108/23] from ovn-kubernetes 43m Normal Started pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Container started 43m Normal Killing pod/payload-pre-processing-e2e-api-b-31e10b-77669586dc-vpx9b Stopping container payload-pre-processing 43m Normal Killing pod/payload-processing-e2e-api-b-31e10b-59678cd5fb-ppdsc Stopping container payload-processing 43m Warning Unhealthy pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Readiness probe failed: Get "http://10.133.0.181:15021/healthz/ready": dial tcp 10.133.0.181:15021: connect: connection refused 43m Normal Created pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Container created 43m Normal Started pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Container started 43m Normal Pulled pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Warning ProbeError pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Readiness probe error: Get "http://10.133.0.181:15021/healthz/ready": dial tcp 10.133.0.181:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Startup probe failed: Get "http://10.133.0.181:15021/healthz/ready": dial tcp 10.133.0.181:15021: connect: connection refused 43m Warning ProbeError pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Startup probe error: Get "http://10.133.0.181:15021/healthz/ready": dial tcp 10.133.0.181:15021: connect: connection refused... 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2a3e01-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2a3e01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Normal AddedLabel gateway/e2e-inf-b-649303 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-649303 43m Warning Unhealthy pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Readiness probe failed: Get "http://10.134.0.105:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 43m Warning ProbeError pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Readiness probe error: Get "http://10.134.0.105:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 43m Normal SuccessfulCreate replicaset/e2e-inf-b-649303-openshift-default-85fbcccd4c Created pod: e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h 43m Normal ScalingReplicaSet deployment/e2e-inf-b-649303-openshift-default Scaled up replica set e2e-inf-b-649303-openshift-default-85fbcccd4c from 0 to 1 43m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw 43m Normal Created pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Container created 43m Normal Started pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Container started 43m Normal Pulled pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Add eth0 [10.134.0.111/23] from ovn-kubernetes 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 4; reason: All metrics below target 43m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled down replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 5 to 4 43m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-fhbvw Stopping container istio-proxy 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-649303 Scaled up replica set payload-processing-e2e-inf-b-649303-5c6ff5544 from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-649303-75547b557f Error creating: pods "payload-pre-processing-e2e-inf-b-649303-75547b557f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-649303: serviceaccount "payload-processing-e2e-inf-b-649303" not found 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-649303-75547b557f Created pod: payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-649303-5c6ff5544 Error creating: pods "payload-processing-e2e-inf-b-649303-5c6ff5544-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-649303: serviceaccount "payload-processing-e2e-inf-b-649303" not found 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-649303-5c6ff5544 Created pod: payload-processing-e2e-inf-b-649303-5c6ff5544-429vh 42m Warning Unhealthy pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Startup probe failed: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-649303 Scaled up replica set payload-pre-processing-e2e-inf-b-649303-75547b557f from 0 to 1 42m Warning ProbeError pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Startup probe error: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused... 42m Normal Started pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Container started 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2a3e01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2a3e01-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Normal Created pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Container created 42m Normal Pulled pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Started pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Container started 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Add eth0 [10.133.0.188/23] from ovn-kubernetes 42m Normal Created pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Container created 42m Normal Pulled pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Add eth0 [10.133.0.189/23] from ovn-kubernetes 42m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default Scaled down replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 4 to 3 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 3; reason: All metrics below target 42m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg 42m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg Stopping container istio-proxy 42m Normal Killing pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf Stopping container istio-proxy 42m Normal Killing pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-n477d Stopping container istio-proxy 42m Normal Killing pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-69n8q Stopping container istio-proxy 42m Normal Started pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Container started 42m Normal Created pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Container created 42m Normal Pulled pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-14e8a4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-14e8a4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning ProbeError pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Startup probe error: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Startup probe failed: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused 42m Normal Killing pod/payload-processing-e2e-api-a-5b9e8e-54bcd7c9c4-k6pjc Stopping container payload-processing 42m Normal Killing pod/payload-pre-processing-e2e-api-a-5b9e8e-5959b68fc5-zljnn Stopping container payload-pre-processing 42m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled down replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 5 to 3 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 3; reason: All metrics below target 42m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 42m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 Stopping container istio-proxy 42m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Stopping container istio-proxy 42m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw 42m Normal SuccessfulCreate replicaset/e2e-inf-a-6af611-openshift-default-6f4cb97d85 Created pod: e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 42m Normal ScalingReplicaSet deployment/e2e-inf-a-6af611-openshift-default Scaled up replica set e2e-inf-a-6af611-openshift-default-6f4cb97d85 from 0 to 1 42m Normal AddedLabel gateway/e2e-inf-a-6af611 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-6af611 42m Normal AddedInterface pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Add eth0 [10.133.0.190/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Readiness probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 42m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw Readiness probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-14e8a4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-14e8a4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95 Error creating: pods "payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-6af611: serviceaccount "payload-processing-e2e-inf-a-6af611" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-6af611 Scaled up replica set payload-processing-e2e-inf-a-6af611-66b99cd54 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-6af611-66b99cd54 Created pod: payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-6af611 Scaled up replica set payload-pre-processing-e2e-inf-a-6af611-66b65b6d95 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95 Created pod: payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-6af611-66b99cd54 Error creating: pods "payload-processing-e2e-inf-a-6af611-66b99cd54-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-6af611: serviceaccount "payload-processing-e2e-inf-a-6af611" not found 42m Normal Killing pod/e2e-api-a-f3160a-openshift-default-67c76678c-77hqr Stopping container istio-proxy 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Add eth0 [10.134.0.118/23] from ovn-kubernetes 42m Normal Created pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Container created 42m Normal Started pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Container started 42m Normal Pulled pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Add eth0 [10.134.0.117/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Container created 42m Normal Started pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Container started 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-11cfa6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-11cfa6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal ScalingReplicaSet deployment/e2e-worker-w2-c2ce75-openshift-default Scaled down replica set e2e-worker-w2-c2ce75-openshift-default-699569fc6c from 3 to 2 42m Normal Killing pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 Stopping container istio-proxy 42m Normal SuccessfulDelete replicaset/e2e-worker-w2-c2ce75-openshift-default-699569fc6c Deleted pod: e2e-worker-w2-c2ce75-openshift-default-699569fc6c-c5tg6 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default New size: 2; reason: All metrics below target 42m Normal Killing pod/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz Stopping container payload-pre-processing 42m Normal Killing pod/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl Stopping container payload-processing 42m Normal Killing pod/payload-pre-processing-e2e-api-b-716b4c-7df4b898c-fc2gh Stopping container payload-pre-processing 42m Normal Killing pod/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q Stopping container payload-processing 42m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled down replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 3 to 2 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 2; reason: All metrics below target 41m Normal Killing pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Deleted pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q 41m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default New size: 4; reason: All metrics below target 41m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xxf2x Stopping container istio-proxy 41m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Readiness probe error: Get "http://10.133.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-gp59q Readiness probe failed: Get "http://10.133.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 41m Normal AddedLabel gateway/e2e-inf-b-6f9071 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-6f9071 41m Normal ScalingReplicaSet deployment/e2e-inf-b-6f9071-openshift-default Scaled up replica set e2e-inf-b-6f9071-openshift-default-657cb56d95 from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-b-6f9071-openshift-default-657cb56d95 Created pod: e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj 41m Normal AddedInterface pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Add eth0 [10.133.0.201/23] from ovn-kubernetes 41m Normal Killing pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Stopping container istio-proxy 41m Normal Killing pod/e2e-api-b-31e10b-openshift-default-5c5846b755-9kgsz Stopping container istio-proxy 41m Normal AddedLabel gateway/e2e-inf-a-80fbd9 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-80fbd9 41m Normal ScalingReplicaSet deployment/e2e-inf-a-80fbd9-openshift-default Scaled up replica set e2e-inf-a-80fbd9-openshift-default-cf4656f6f from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-a-80fbd9-openshift-default-cf4656f6f Created pod: e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-6f9071 Scaled up replica set payload-pre-processing-e2e-inf-b-6f9071-7545c84995 from 0 to 1 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-6f9071-d78bbfb84 Error creating: pods "payload-processing-e2e-inf-b-6f9071-d78bbfb84-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-6f9071: serviceaccount "payload-processing-e2e-inf-b-6f9071" not found 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-6f9071-7545c84995 Created pod: payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-6f9071-d78bbfb84 Created pod: payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-6f9071 Scaled up replica set payload-processing-e2e-inf-b-6f9071-d78bbfb84 from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-6f9071-7545c84995 Error creating: pods "payload-pre-processing-e2e-inf-b-6f9071-7545c84995-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-6f9071: serviceaccount "payload-processing-e2e-inf-b-6f9071" not found 41m Normal Started pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Container started 41m Normal AddedInterface pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Add eth0 [10.134.0.120/23] from ovn-kubernetes 41m Normal Pulled pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Container started 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Add eth0 [10.133.0.203/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Container created 41m Warning ProbeError pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Readiness probe error: HTTP probe failed with statuscode: 503... 41m Warning Unhealthy pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn Readiness probe failed: HTTP probe failed with statuscode: 503 41m Normal Created pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Container created 41m Normal AddedInterface pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Add eth0 [10.133.0.202/23] from ovn-kubernetes 41m Normal Started pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Container started 41m Warning ProbeError pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Readiness probe error: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused... 41m Normal Pulled pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Container created 41m Warning Unhealthy pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Readiness probe failed: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79 Error creating: pods "payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-80fbd9: serviceaccount "payload-processing-e2e-inf-a-80fbd9" not found 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-80fbd9 Scaled up replica set payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79 from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79 Created pod: payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-80fbd9-866f75d5f8 Error creating: pods "payload-processing-e2e-inf-a-80fbd9-866f75d5f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-80fbd9: serviceaccount "payload-processing-e2e-inf-a-80fbd9" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-80fbd9 Scaled up replica set payload-processing-e2e-inf-a-80fbd9-866f75d5f8 from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-80fbd9-866f75d5f8 Created pod: payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 41m Normal Created pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Container created 41m Normal Pulled pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Container created 41m Normal Pulled pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Add eth0 [10.133.0.204/23] from ovn-kubernetes 41m Normal Started pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Container started 41m Normal Started pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Container started 41m Normal AddedInterface pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Add eth0 [10.133.0.205/23] from ovn-kubernetes 41m Warning ProbeError pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Startup probe error: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused... 41m Normal Started pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Container started 41m Warning Unhealthy pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Startup probe failed: Get "http://10.133.0.187:15021/healthz/ready": dial tcp 10.133.0.187:15021: connect: connection refused 41m Normal Pulled pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Created pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Container created 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-649303-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-649303-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-4a8ee1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-4a8ee1-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning ProbeError pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Startup probe error: Get "http://10.134.0.111:15021/healthz/ready": dial tcp 10.134.0.111:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Startup probe failed: Get "http://10.134.0.111:15021/healthz/ready": dial tcp 10.134.0.111:15021: connect: connection refused 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-6f9071-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-6f9071-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-4a8ee1-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-4a8ee1-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-649303-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-649303-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 3; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled down replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 5 to 3 40m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-sdwk8 40m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh 40m Normal Killing pod/e2e-api-a-5b9e8e-openshift-default-77bfd57d69-29c8l Stopping container istio-proxy 40m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Readiness probe error: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 40m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-5vwbh Readiness probe failed: Get "http://10.132.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 40m Normal Created pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Container created 40m Normal Started pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Container started 40m Normal Pulled pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Killing pod/payload-pre-processing-e2e-api-b-14e8a4-6d4c8497f-mm85g Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn Stopping container payload-processing 40m Normal Killing pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw Stopping container istio-proxy 40m Warning ProbeError pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Startup probe error: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Startup probe failed: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused 40m Normal Killing pod/payload-pre-processing-e2e-api-b-11cfa6-645cb4fb75-l24ps Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk Stopping container payload-processing 40m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j Stopping container istio-proxy 40m Normal AddedLabel gateway/e2e-inf-b-cd2d2e Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-cd2d2e 40m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz 40m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zzp4j 40m Normal ScalingReplicaSet deployment/e2e-inf-b-cd2d2e-openshift-default Scaled up replica set e2e-inf-b-cd2d2e-openshift-default-57fd5565d9 from 0 to 1 40m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled down replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 7 to 5 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 5; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nr5bz Stopping container istio-proxy 40m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l Stopping container istio-proxy 40m Normal AddedInterface pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Add eth0 [10.134.0.121/23] from ovn-kubernetes 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-7lx8l 40m Normal SuccessfulCreate replicaset/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9 Created pod: e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-6af611-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-6af611-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-cd2d2e Scaled up replica set payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b from 0 to 1 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b Error creating: pods "payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-cd2d2e: serviceaccount "payload-processing-e2e-inf-b-cd2d2e" not found 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b Created pod: payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c Error creating: pods "payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-cd2d2e: serviceaccount "payload-processing-e2e-inf-b-cd2d2e" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c Created pod: payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-cd2d2e Scaled up replica set payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c from 0 to 1 40m Normal Pulled pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Add eth0 [10.134.0.122/23] from ovn-kubernetes 40m Normal Pulled pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Add eth0 [10.134.0.123/23] from ovn-kubernetes 40m Normal Started pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Container started 40m Normal Created pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Container created 40m Normal Started pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Container started 40m Normal Created pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Container created 40m Normal Killing pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-inf-a-618c72-openshift-default Scaled up replica set e2e-inf-a-618c72-openshift-default-7d44fcfdf7 from 0 to 1 40m Normal AddedInterface pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Add eth0 [10.134.0.124/23] from ovn-kubernetes 40m Normal SuccessfulCreate replicaset/e2e-inf-a-618c72-openshift-default-7d44fcfdf7 Created pod: e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v 40m Normal AddedLabel gateway/e2e-inf-a-618c72 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-618c72 40m Warning ProbeError pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-618c72 Scaled up replica set payload-processing-e2e-inf-a-618c72-5bbdd8858 from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-618c72-5bbdd8858 Created pod: payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5 Error creating: pods "payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-618c72: serviceaccount "payload-processing-e2e-inf-a-618c72" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5 Created pod: payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-618c72 Scaled up replica set payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5 from 0 to 1 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-6af611-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-618c72-5bbdd8858 Error creating: pods "payload-processing-e2e-inf-a-618c72-5bbdd8858-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-618c72: serviceaccount "payload-processing-e2e-inf-a-618c72" not found 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-6af611-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal Pulled pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Add eth0 [10.134.0.126/23] from ovn-kubernetes 40m Normal AddedInterface pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Add eth0 [10.134.0.125/23] from ovn-kubernetes 40m Normal Pulled pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Container created 40m Normal Started pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Container started 40m Normal Created pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Container created 40m Normal Started pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Container started 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-2a3e01-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 40m Normal ScalingReplicaSet deployment/e2e-inf-a-2a3e01-openshift-default Scaled up replica set e2e-inf-a-2a3e01-openshift-default-56dfc4f47 from 1 to 2 40m Normal SuccessfulCreate replicaset/e2e-inf-a-2a3e01-openshift-default-56dfc4f47 Created pod: e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw 40m Normal AddedInterface pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Add eth0 [10.134.0.127/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Container created 40m Normal Started pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Container started 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-618c72-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-618c72-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning ProbeError pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Startup probe error: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Startup probe failed: Get "http://10.134.0.127:15021/healthz/ready": dial tcp 10.134.0.127:15021: connect: connection refused 39m Normal Pulled pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Container created 39m Normal Started pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Container started 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-80fbd9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-80fbd9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning Unhealthy pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Startup probe failed: Get "http://10.133.0.202:15021/healthz/ready": dial tcp 10.133.0.202:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Startup probe error: Get "http://10.133.0.202:15021/healthz/ready": dial tcp 10.133.0.202:15021: connect: connection refused... 39m Normal Started pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Container started 39m Normal Created pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Container created 39m Normal Pulled pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal ScalingReplicaSet deployment/e2e-inf-b-bec983-openshift-default Scaled up replica set e2e-inf-b-bec983-openshift-default-f5ff49c7c from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-inf-b-bec983-openshift-default-f5ff49c7c Created pod: e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m 39m Normal AddedLabel gateway/e2e-inf-b-bec983 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-bec983 39m Normal AddedInterface pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Add eth0 [10.133.0.208/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-bec983-7d8dddc58 Created pod: payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj 39m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-bec983-7d8dddc58 Error creating: pods "payload-processing-e2e-inf-b-bec983-7d8dddc58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-bec983: serviceaccount "payload-processing-e2e-inf-b-bec983" not found 39m Warning ProbeError pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Startup probe error: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Startup probe failed: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-bec983 Scaled up replica set payload-pre-processing-e2e-inf-b-bec983-66679fd98c from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-bec983-66679fd98c Created pod: payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-bec983-66679fd98c Error creating: pods "payload-pre-processing-e2e-inf-b-bec983-66679fd98c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-bec983: serviceaccount "payload-processing-e2e-inf-b-bec983" not found 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-bec983 Scaled up replica set payload-processing-e2e-inf-b-bec983-7d8dddc58 from 0 to 1 39m Normal Created pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Container created 39m Normal Started pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Container started 39m Normal Started pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Container started 39m Normal Created pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Container created 39m Normal AddedInterface pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Add eth0 [10.134.0.129/23] from ovn-kubernetes 39m Normal Pulled pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Pulled pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Add eth0 [10.134.0.128/23] from ovn-kubernetes 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-6f9071-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-6f9071-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-80fbd9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-80fbd9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-bec983-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-bec983-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Killing pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng Stopping container istio-proxy 39m Normal AddedLabel gateway/e2e-inf-a-60a9a6 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-60a9a6 39m Normal AddedInterface pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Add eth0 [10.133.0.209/23] from ovn-kubernetes 39m Normal SuccessfulCreate replicaset/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c Created pod: e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d 39m Normal ScalingReplicaSet deployment/e2e-inf-a-60a9a6-openshift-default Scaled up replica set e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c from 0 to 1 39m Normal Created pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Container created 39m Normal Started pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Container started 39m Normal Pulled pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-60a9a6 Scaled up replica set payload-processing-e2e-inf-a-60a9a6-d569c84f4 from 0 to 1 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655 Error creating: pods "payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-60a9a6: serviceaccount "payload-processing-e2e-inf-a-60a9a6" not found 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-60a9a6-d569c84f4 Error creating: pods "payload-processing-e2e-inf-a-60a9a6-d569c84f4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-60a9a6: serviceaccount "payload-processing-e2e-inf-a-60a9a6" not found 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-60a9a6 Scaled up replica set payload-pre-processing-e2e-inf-a-60a9a6-847b54b655 from 0 to 1 38m Normal Created pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Container created 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655 Created pod: payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Add eth0 [10.134.0.130/23] from ovn-kubernetes 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-60a9a6-d569c84f4 Created pod: payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql 38m Normal Started pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Container started 38m Normal Started pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Container started 38m Normal Created pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Container created 38m Normal Pulled pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Add eth0 [10.134.0.131/23] from ovn-kubernetes 38m Warning Unhealthy pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Startup probe failed: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Startup probe error: Get "http://10.134.0.121:15021/healthz/ready": dial tcp 10.134.0.121:15021: connect: connection refused... 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-6f9071-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 38m Normal SuccessfulCreate replicaset/e2e-inf-b-6f9071-openshift-default-657cb56d95 Created pod: e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr 38m Normal ScalingReplicaSet deployment/e2e-inf-b-6f9071-openshift-default Scaled up replica set e2e-inf-b-6f9071-openshift-default-657cb56d95 from 1 to 2 38m Normal Started pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Container started 38m Normal AddedInterface pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Add eth0 [10.134.0.133/23] from ovn-kubernetes 38m Normal Killing pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Stopping container istio-proxy 38m Normal Created pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Container created 38m Normal Pulled pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Warning Unhealthy pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Readiness probe failed: HTTP probe failed with statuscode: 503 38m Normal ScalingReplicaSet deployment/e2e-inf-a-2b8461-openshift-default Scaled up replica set e2e-inf-a-2b8461-openshift-default-7c444f8877 from 0 to 1 38m Normal SuccessfulCreate replicaset/e2e-inf-a-2b8461-openshift-default-7c444f8877 Created pod: e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk 38m Normal AddedLabel gateway/e2e-inf-a-2b8461 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-2b8461 38m Warning ProbeError pod/e2e-api-b-11cfa6-openshift-default-6dd9b4cb7-887rf Readiness probe error: HTTP probe failed with statuscode: 503... 38m Normal AddedInterface pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Add eth0 [10.134.0.134/23] from ovn-kubernetes 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-cd2d2e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-cd2d2e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning ProbeError pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Startup probe error: Get "http://10.134.0.133:15021/healthz/ready": dial tcp 10.134.0.133:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Startup probe failed: Get "http://10.134.0.133:15021/healthz/ready": dial tcp 10.134.0.133:15021: connect: connection refused 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-2b8461 Scaled up replica set payload-processing-e2e-inf-a-2b8461-86cb99fc6b from 0 to 1 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-2b8461-86cb99fc6b Error creating: pods "payload-processing-e2e-inf-a-2b8461-86cb99fc6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2b8461: serviceaccount "payload-processing-e2e-inf-a-2b8461" not found 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-2b8461-86cb99fc6b Created pod: payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-2b8461 Scaled up replica set payload-pre-processing-e2e-inf-a-2b8461-56d688db75 from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-2b8461-56d688db75 Created pod: payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-2b8461-56d688db75 Error creating: pods "payload-pre-processing-e2e-inf-a-2b8461-56d688db75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-2b8461: serviceaccount "payload-processing-e2e-inf-a-2b8461" not found 38m Normal Started pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Container started 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Add eth0 [10.133.0.211/23] from ovn-kubernetes 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Add eth0 [10.133.0.212/23] from ovn-kubernetes 38m Normal Started pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Container started 38m Normal Created pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Container created 38m Normal Pulled pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Container created 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Warning ProbeError pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Readiness probe error: Get "http://10.134.0.124:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 38m Warning Unhealthy pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Readiness probe failed: Get "http://10.134.0.124:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 38m Normal Started pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Container started 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-cd2d2e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal Pulled pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal Created pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Container created 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-cd2d2e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal AddedLabel gateway/e2e-inf-b-619b9b Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-619b9b 38m Normal ScalingReplicaSet deployment/e2e-inf-b-619b9b-openshift-default Scaled up replica set e2e-inf-b-619b9b-openshift-default-65488dc9d7 from 0 to 1 38m Normal SuccessfulCreate replicaset/e2e-inf-b-619b9b-openshift-default-65488dc9d7 Created pod: e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 38m Normal AddedInterface pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Add eth0 [10.132.0.154/23] from ovn-kubernetes 38m Warning Unhealthy pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Startup probe failed: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Startup probe error: Get "http://10.134.0.124:15021/healthz/ready": dial tcp 10.134.0.124:15021: connect: connection refused... 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-619b9b Scaled up replica set payload-processing-e2e-inf-b-619b9b-5585886b4d from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-619b9b Scaled up replica set payload-pre-processing-e2e-inf-b-619b9b-7c8755987f from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-619b9b-5585886b4d Created pod: payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f Created pod: payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f Error creating: pods "payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-619b9b: serviceaccount "payload-processing-e2e-inf-b-619b9b" not found 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-619b9b-5585886b4d Error creating: pods "payload-processing-e2e-inf-b-619b9b-5585886b4d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-619b9b: serviceaccount "payload-processing-e2e-inf-b-619b9b" not found 38m Normal AddedInterface pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Add eth0 [10.134.0.136/23] from ovn-kubernetes 38m Normal Created pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Container created 38m Normal Created pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Container created 38m Normal Started pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Container started 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Add eth0 [10.134.0.135/23] from ovn-kubernetes 38m Normal Pulled pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Started pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Container started 38m Normal Pulled pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Stopping container istio-proxy 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-dd90cb-openshift-default New size: 2; reason: All metrics below target 38m Normal ScalingReplicaSet deployment/e2e-worker-w6-dd90cb-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-dd90cb-openshift-default-597f54ff7 from 3 to 2 38m Normal SuccessfulDelete replicaset/e2e-worker-w6-dd90cb-openshift-default-597f54ff7 Deleted pod: e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-c90b9e-openshift-default New size: 2; reason: All metrics below target 38m Normal ScalingReplicaSet deployment/e2e-worker-w0-c90b9e-openshift-default Scaled down replica set e2e-worker-w0-c90b9e-openshift-default-778d7f78bf from 3 to 2 38m Normal SuccessfulDelete replicaset/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf Deleted pod: e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj 38m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Stopping container istio-proxy 38m Warning Unhealthy pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Readiness probe failed: HTTP probe failed with statuscode: 503 38m Warning ProbeError pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-svdj2 Readiness probe error: HTTP probe failed with statuscode: 503... 38m Warning ProbeError pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Readiness probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-69zrj Readiness probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-618c72-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-618c72-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal Started pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Container started 37m Normal Created pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Container created 37m Normal Pulled pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Warning ProbeError pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Startup probe error: Get "http://10.133.0.208:15021/healthz/ready": dial tcp 10.133.0.208:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Startup probe failed: Get "http://10.133.0.208:15021/healthz/ready": dial tcp 10.133.0.208:15021: connect: connection refused 37m Normal Killing pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Stopping container payload-processing 37m Normal Created pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Container created 37m Normal Pulled pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Container started 37m Normal Killing pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Stopping container payload-pre-processing 37m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-6af611-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 37m Normal SuccessfulCreate replicaset/e2e-inf-a-6af611-openshift-default-6f4cb97d85 Created pod: e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz 37m Normal ScalingReplicaSet deployment/e2e-inf-a-6af611-openshift-default Scaled up replica set e2e-inf-a-6af611-openshift-default-6f4cb97d85 from 1 to 2 37m Warning ProbeError pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Startup probe error: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Startup probe failed: Get "http://10.134.0.134:15021/healthz/ready": dial tcp 10.134.0.134:15021: connect: connection refused 37m Normal Created pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Container created 37m Normal AddedInterface pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Add eth0 [10.132.0.161/23] from ovn-kubernetes 37m Normal Pulled pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Started pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Container started 37m Warning Unhealthy pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Startup probe failed: Get "http://10.132.0.161:15021/healthz/ready": dial tcp 10.132.0.161:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Startup probe error: Get "http://10.132.0.161:15021/healthz/ready": dial tcp 10.132.0.161:15021: connect: connection refused... 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-bec983-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-bec983-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Normal AddedLabel gateway/e2e-inf-b-f3338e Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-f3338e 37m Normal AddedInterface pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Add eth0 [10.132.0.162/23] from ovn-kubernetes 37m Normal SuccessfulCreate replicaset/e2e-inf-b-f3338e-openshift-default-786cf5c989 Created pod: e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k 37m Warning ProbeError pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Readiness probe error: Get "http://10.133.0.209:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 37m Warning Unhealthy pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Readiness probe failed: Get "http://10.133.0.209:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 37m Normal ScalingReplicaSet deployment/e2e-inf-b-f3338e-openshift-default Scaled up replica set e2e-inf-b-f3338e-openshift-default-786cf5c989 from 0 to 1 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2b8461-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2b8461-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp Stopping container istio-proxy 36m Normal Started pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Container started 36m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled down replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 5 to 4 36m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-9zzfp 36m Normal Created pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Container created 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 4; reason: All metrics below target 36m Normal Pulled pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt 36m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 10 to 8 36m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-7462m 36m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-7462m Stopping container istio-proxy 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-7mx6s 36m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Stopping container istio-proxy 36m Warning Unhealthy pod/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-2a3e01-67b866fbcf-hwjnq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979 Created pod: payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979 Error creating: pods "payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-f3338e: serviceaccount "payload-processing-e2e-inf-b-f3338e" not found 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-f3338e Scaled up replica set payload-processing-e2e-inf-b-f3338e-6c57db899d from 0 to 1 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-f3338e-6c57db899d Error creating: pods "payload-processing-e2e-inf-b-f3338e-6c57db899d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-f3338e: serviceaccount "payload-processing-e2e-inf-b-f3338e" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-f3338e Scaled up replica set payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979 from 0 to 1 36m Normal Created pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Container created 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Add eth0 [10.134.0.137/23] from ovn-kubernetes 36m Normal Pulled pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7mx6s Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal Started pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Container started 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-f3338e-6c57db899d Created pod: payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd 36m Normal Created pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Container created 36m Normal Started pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Container started 36m Normal Pulled pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Add eth0 [10.133.0.219/23] from ovn-kubernetes 36m Warning Unhealthy pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Startup probe failed: Get "http://10.133.0.209:15021/healthz/ready": dial tcp 10.133.0.209:15021: connect: connection refused 36m Warning ProbeError pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Startup probe error: Get "http://10.133.0.209:15021/healthz/ready": dial tcp 10.133.0.209:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Readiness probe failed: Get "http://10.133.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-xk5wt Readiness probe error: Get "http://10.133.0.96:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-60a9a6-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-60a9a6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal Killing pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Stopping container istio-proxy 36m Warning Unhealthy pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Stopping container payload-processing 36m Warning Unhealthy pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-7lfn6 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Normal Killing pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Stopping container payload-pre-processing 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-2b8461-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-2b8461-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Stopping container istio-proxy 36m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Stopping container istio-proxy 36m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: All metrics below target 36m Normal AddedLabel gateway/e2e-inf-b-a5e47c Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-a5e47c 36m Normal ScalingReplicaSet deployment/e2e-inf-b-a5e47c-openshift-default Scaled up replica set e2e-inf-b-a5e47c-openshift-default-66d7979c75 from 0 to 1 36m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 8 to 6 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-qg8wr 36m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-q8w96 36m Normal AddedInterface pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Add eth0 [10.133.0.221/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-inf-b-a5e47c-openshift-default-66d7979c75 Created pod: e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl 36m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-q8w96 Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-qg8wr Readiness probe failed: HTTP probe failed with statuscode: 503 36m Normal Killing pod/payload-processing-e2e-inf-b-649303-5c6ff5544-429vh Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Stopping container payload-pre-processing 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-a5e47c Scaled up replica set payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47 from 0 to 1 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-a5e47c Scaled up replica set payload-processing-e2e-inf-b-a5e47c-6cf788968 from 0 to 1 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-a5e47c-6cf788968 Error creating: pods "payload-processing-e2e-inf-b-a5e47c-6cf788968-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-a5e47c: serviceaccount "payload-processing-e2e-inf-b-a5e47c" not found 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-a5e47c-6cf788968 Created pod: payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47 Error creating: pods "payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-a5e47c: serviceaccount "payload-processing-e2e-inf-b-a5e47c" not found 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-60a9a6-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47 Created pod: payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-60a9a6-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal Started pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Container started 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Add eth0 [10.133.0.222/23] from ovn-kubernetes 36m Normal AddedInterface pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Add eth0 [10.133.0.223/23] from ovn-kubernetes 36m Normal Pulled pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Pulled pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Started pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Container started 36m Normal Created pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Container created 36m Normal Created pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Container created 36m Normal Started pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Container started 36m Normal Created pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Container created 36m Normal Pulled pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-619b9b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-619b9b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-q8n9f 36m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Warning ProbeError pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Startup probe error: Get "http://10.132.0.154:15021/healthz/ready": dial tcp 10.132.0.154:15021: connect: connection refused... 36m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-q8n9f Stopping container discovery 36m Warning Unhealthy pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Startup probe failed: Get "http://10.132.0.154:15021/healthz/ready": dial tcp 10.132.0.154:15021: connect: connection refused 36m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-649303-75547b557f-wql6w Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Warning Unhealthy pod/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 36m Normal Killing pod/e2e-inf-a-4a8ee1-openshift-default-566547cf95-bfn6v Stopping container istio-proxy 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-bec983-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal ScalingReplicaSet deployment/e2e-inf-b-bec983-openshift-default Scaled up replica set e2e-inf-b-bec983-openshift-default-f5ff49c7c from 1 to 2 36m Normal Started pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Container started 36m Normal Pulled pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Add eth0 [10.134.0.138/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-inf-b-bec983-openshift-default-f5ff49c7c Created pod: e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv 36m Normal Created pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Container created 36m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v 36m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-4pt6v Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled down replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 3 to 2 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 2; reason: All metrics below target 36m Warning ProbeError pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Startup probe error: Get "http://10.134.0.138:15021/healthz/ready": dial tcp 10.134.0.138:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Startup probe failed: Get "http://10.134.0.138:15021/healthz/ready": dial tcp 10.134.0.138:15021: connect: connection refused 36m Normal AddedInterface pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Add eth0 [10.134.0.140/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-inf-a-80fbd9-openshift-default-cf4656f6f Created pod: e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 36m Normal ScalingReplicaSet deployment/e2e-inf-a-80fbd9-openshift-default Scaled up replica set e2e-inf-a-80fbd9-openshift-default-cf4656f6f from 1 to 2 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-80fbd9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal Created pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Container created 36m Normal Killing pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Stopping container payload-pre-processing 36m Normal Killing pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Stopping container payload-processing 36m Normal Pulled pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Container started 36m Warning ProbeError pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Startup probe error: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Startup probe failed: Get "http://10.134.0.140:15021/healthz/ready": dial tcp 10.134.0.140:15021: connect: connection refused 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-619b9b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-619b9b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-6f9071-7545c84995-w5kwk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning Unhealthy pod/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Normal Killing pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h Stopping container istio-proxy 35m Normal Killing pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-b82tj Stopping container istio-proxy 35m Normal Killing pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr Stopping container istio-proxy 35m Normal Started pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Container started 35m Normal Pulled pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Pulled pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Created pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Container created 35m Normal Created pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Container created 35m Normal Started pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Container started 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-a5e47c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-a5e47c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning Unhealthy pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Startup probe failed: Get "http://10.132.0.162:15021/healthz/ready": dial tcp 10.132.0.162:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Startup probe error: Get "http://10.132.0.162:15021/healthz/ready": dial tcp 10.132.0.162:15021: connect: connection refused... 34m Warning ProbeError pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Startup probe error: Get "http://10.133.0.221:15021/healthz/ready": dial tcp 10.133.0.221:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Startup probe failed: Get "http://10.133.0.221:15021/healthz/ready": dial tcp 10.133.0.221:15021: connect: connection refused 34m Normal Killing pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Stopping container payload-processing 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-f3338e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-f3338e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning Unhealthy pod/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-a5e47c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-a5e47c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Normal SuccessfulCreate replicaset/e2e-inf-a-618c72-openshift-default-7d44fcfdf7 Created pod: e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv 34m Normal AddedInterface pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Add eth0 [10.133.0.234/23] from ovn-kubernetes 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-618c72-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 34m Normal ScalingReplicaSet deployment/e2e-inf-a-618c72-openshift-default Scaled up replica set e2e-inf-a-618c72-openshift-default-7d44fcfdf7 from 1 to 2 34m Normal Created pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Container created 34m Normal Started pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Container started 34m Normal Pulled pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Killing pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-t9fmz Stopping container istio-proxy 34m Normal Killing pod/e2e-inf-a-6af611-openshift-default-6f4cb97d85-n9648 Stopping container istio-proxy 34m Warning Unhealthy pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Startup probe failed: Get "http://10.133.0.234:15021/healthz/ready": dial tcp 10.133.0.234:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Startup probe error: Get "http://10.133.0.234:15021/healthz/ready": dial tcp 10.133.0.234:15021: connect: connection refused... 34m Normal Killing pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Stopping container payload-processing 33m Normal Started pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Container started 33m Normal Created pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Container created 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-60a9a6-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 33m Normal ScalingReplicaSet deployment/e2e-inf-a-60a9a6-openshift-default Scaled up replica set e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c from 1 to 2 33m Normal Pulled pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal SuccessfulCreate replicaset/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c Created pod: e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 33m Normal AddedInterface pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Add eth0 [10.134.0.146/23] from ovn-kubernetes 33m Warning Unhealthy pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Startup probe failed: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused 33m Warning ProbeError pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Startup probe error: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused... 33m Warning Unhealthy pod/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal Killing pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Stopping container payload-processing 33m Normal Killing pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Stopping container payload-pre-processing 33m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-8982g Stopping container istio-proxy 33m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 6 to 5 33m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 33m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-8982g 33m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-8982g Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-8982g Readiness probe failed: HTTP probe failed with statuscode: 503 33m Normal Killing pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb Stopping container istio-proxy 33m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc Created pod: e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx 33m Normal AddedLabel gateway/e2e-shared-a-w4-51288b Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w4-51288b 33m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-51288b-openshift-default Scaled up replica set e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc from 0 to 1 33m Normal AddedInterface pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Add eth0 [10.134.0.147/23] from ovn-kubernetes 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 3; reason: All metrics below target 33m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz 33m Warning Unhealthy pod/payload-processing-e2e-inf-a-80fbd9-866f75d5f8-wgcr4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Stopping container istio-proxy 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-80fbd9-5c485b6b79-nt9k4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default Scaled down replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 4 to 3 33m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-nhctz Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667 Created pod: payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w4-51288b Scaled up replica set payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8 from 0 to 1 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8 Error creating: pods "payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-51288b: serviceaccount "payload-processing-e2e-shared-a-w4-51288b" not found 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8 Created pod: payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w4-51288b Scaled up replica set payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667 from 0 to 1 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667 Error creating: pods "payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-51288b: serviceaccount "payload-processing-e2e-shared-a-w4-51288b" not found 33m Normal AddedInterface pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Add eth0 [10.133.0.242/23] from ovn-kubernetes 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Add eth0 [10.133.0.241/23] from ovn-kubernetes 33m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Created pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Container created 33m Normal Pulled pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Started pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Container started 33m Normal Created pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Container created 33m Normal Started pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Container started 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-2b8461-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 32m Normal Pulled pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal AddedInterface pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Add eth0 [10.133.0.243/23] from ovn-kubernetes 32m Normal Created pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Container created 32m Normal SuccessfulCreate replicaset/e2e-inf-a-2b8461-openshift-default-7c444f8877 Created pod: e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 32m Normal Started pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Container started 32m Normal ScalingReplicaSet deployment/e2e-inf-a-2b8461-openshift-default Scaled up replica set e2e-inf-a-2b8461-openshift-default-7c444f8877 from 1 to 2 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-a5e47c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w2-acd125-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 32m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-acd125-openshift-default Scaled up replica set e2e-shared-a-w2-acd125-openshift-default-688b6f7db7 from 1 to 2 32m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7 Created pod: e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 32m Normal ScalingReplicaSet deployment/e2e-inf-b-a5e47c-openshift-default Scaled up replica set e2e-inf-b-a5e47c-openshift-default-66d7979c75 from 1 to 2 32m Normal SuccessfulCreate replicaset/e2e-inf-b-a5e47c-openshift-default-66d7979c75 Created pod: e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 32m Normal AddedInterface pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Add eth0 [10.134.0.148/23] from ovn-kubernetes 32m Normal Created pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Container created 32m Normal Created pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Container created 32m Normal Started pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Container started 32m Normal Pulled pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Container started 32m Normal Pulled pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal AddedInterface pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Add eth0 [10.134.0.149/23] from ovn-kubernetes 32m Warning ProbeError pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Startup probe error: Get "http://10.133.0.243:15021/healthz/ready": dial tcp 10.133.0.243:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Startup probe failed: Get "http://10.133.0.243:15021/healthz/ready": dial tcp 10.133.0.243:15021: connect: connection refused 32m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 32m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Created pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv 32m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0e874e-openshift-default Scaled up replica set e2e-shared-b-w5-0e874e-openshift-default-594f895d85 from 1 to 2 32m Warning Unhealthy pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Startup probe failed: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused 32m Warning ProbeError pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Startup probe error: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused... 32m Normal Created pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Container created 32m Warning Unhealthy pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Startup probe failed: Get "http://10.134.0.149:15021/healthz/ready": dial tcp 10.134.0.149:15021: connect: connection refused 32m Warning ProbeError pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Startup probe error: Get "http://10.134.0.149:15021/healthz/ready": dial tcp 10.134.0.149:15021: connect: connection refused... 32m Normal Pulled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 32m Normal Started pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Container started 32m Normal AddedInterface pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Add eth0 [10.132.0.163/23] from ovn-kubernetes 32m Normal Killing pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Stopping container payload-pre-processing 32m Normal Killing pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-lwl9f Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 Stopping container istio-proxy 32m Normal Killing pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Stopping container payload-processing 32m Warning ProbeError pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Startup probe error: Get "http://10.132.0.163:15021/healthz/ready": dial tcp 10.132.0.163:15021: connect: connection refused... 32m Warning Unhealthy pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Startup probe failed: Get "http://10.132.0.163:15021/healthz/ready": dial tcp 10.132.0.163:15021: connect: connection refused 32m Normal Killing pod/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Stopping container payload-pre-processing 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning FailedCreate replicaset/e2e-inf-a-618c72-openshift-default-7d44fcfdf7 Error creating: pods "e2e-inf-a-618c72-openshift-default-7d44fcfdf7-" is forbidden: error looking up service account openshift-ingress/e2e-inf-a-618c72-openshift-default: serviceaccount "e2e-inf-a-618c72-openshift-default" not found 32m Normal Killing pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-zx8nv Stopping container istio-proxy 32m Normal ScalingReplicaSet deployment/e2e-inf-a-618c72-openshift-default Scaled up replica set e2e-inf-a-618c72-openshift-default-7d44fcfdf7 from 0 to 1 32m Normal Killing pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v Stopping container istio-proxy 32m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 5 to 4 32m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-tvm7w 32m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Stopping container istio-proxy 32m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 32m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-tvm7w Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal Killing pod/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Stopping container payload-pre-processing 31m Normal Killing pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-pcr5m Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv Stopping container istio-proxy 31m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Normal Created pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Container created 31m Normal Started pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Container started 31m Normal Pulled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-627dfa-openshift-default Scaled up replica set e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 from 0 to 1 31m Normal AddedLabel gateway/e2e-shared-b-w4-627dfa Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w4-627dfa 31m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 Created pod: e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj 31m Warning Unhealthy pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Startup probe failed: Get "http://10.134.0.147:15021/healthz/ready": dial tcp 10.134.0.147:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Startup probe error: Get "http://10.134.0.147:15021/healthz/ready": dial tcp 10.134.0.147:15021: connect: connection refused... 31m Normal AddedInterface pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Add eth0 [10.134.0.152/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Created pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 31m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled up replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 1 to 2 31m Normal AddedInterface pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Add eth0 [10.134.0.154/23] from ovn-kubernetes 31m Normal Created pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Container created 31m Normal Started pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Container started 31m Normal Pulled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w4-627dfa Scaled up replica set payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd from 0 to 1 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c Error creating: pods "payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa: serviceaccount "payload-processing-e2e-shared-b-w4-627dfa" not found 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c Created pod: payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd Error creating: pods "payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-627dfa: serviceaccount "payload-processing-e2e-shared-b-w4-627dfa" not found 31m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Add eth0 [10.133.0.252/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd Created pod: payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w4-627dfa Scaled up replica set payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c from 0 to 1 31m Normal AddedInterface pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Add eth0 [10.133.0.253/23] from ovn-kubernetes 31m Normal Created pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Container created 31m Normal Started pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Container started 31m Normal Pulled pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Container started 31m Normal Created pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Container created 31m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Startup probe failed: Get "http://10.134.0.154:15021/healthz/ready": dial tcp 10.134.0.154:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Startup probe error: Get "http://10.134.0.154:15021/healthz/ready": dial tcp 10.134.0.154:15021: connect: connection refused... 31m Normal Killing pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv Stopping container istio-proxy 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-09db16-openshift-default New size: 1; reason: All metrics below target 31m Normal SuccessfulDelete replicaset/e2e-worker-w5-7a3603-openshift-default-b6494fbc9 Deleted pod: e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs 31m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Stopping container istio-proxy 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-7a3603-openshift-default New size: 2; reason: All metrics below target 31m Normal ScalingReplicaSet deployment/e2e-worker-w3-09db16-openshift-default Scaled down replica set e2e-worker-w3-09db16-openshift-default-5d4f7c499 from 2 to 1 31m Normal SuccessfulDelete replicaset/e2e-worker-w3-09db16-openshift-default-5d4f7c499 Deleted pod: e2e-worker-w3-09db16-openshift-default-5d4f7c499-9prhv 31m Normal ScalingReplicaSet deployment/e2e-worker-w5-7a3603-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w5-7a3603-openshift-default-b6494fbc9 from 3 to 2 31m Normal SuccessfulDelete replicaset/e2e-worker-w1-8b46c9-openshift-default-6c8db6467 Deleted pod: e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 31m Normal ScalingReplicaSet deployment/e2e-worker-w1-8b46c9-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-8b46c9-openshift-default-6c8db6467 from 2 to 1 31m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Stopping container istio-proxy 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-8b46c9-openshift-default (combined from similar events): New size: 1; reason: All metrics below target 31m Warning Unhealthy pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-xsbqs Readiness probe error: HTTP probe failed with statuscode: 503... 31m Normal AddedInterface pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Add eth0 [10.133.0.254/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-96ab73-openshift-default-586f58856c Created pod: e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt 31m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-96ab73-openshift-default Scaled up replica set e2e-shared-a-w0-96ab73-openshift-default-586f58856c from 0 to 1 31m Normal AddedLabel gateway/e2e-shared-a-w0-96ab73 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-96ab73 31m Warning Unhealthy pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-inf-b-619b9b-openshift-default-65488dc9d7-sxrl8 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4 Error creating: pods "payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73: serviceaccount "payload-processing-e2e-shared-a-w0-96ab73" not found 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-96ab73 Scaled up replica set payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67 from 0 to 1 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-96ab73 Scaled up replica set payload-processing-e2e-shared-a-w0-96ab73-d779d96b4 from 0 to 1 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67 Error creating: pods "payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73: serviceaccount "payload-processing-e2e-shared-a-w0-96ab73" not found 31m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Container created 31m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Add eth0 [10.134.0.155/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4 Created pod: payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb 31m Normal Started pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Container started 31m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Container started 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Add eth0 [10.134.0.156/23] from ovn-kubernetes 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67 Created pod: payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph 31m Normal Pulled pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Created pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Container created 31m Normal Killing pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql Stopping container payload-processing 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-51288b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-51288b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal ScalingReplicaSet deployment/e2e-worker-w2-c2ce75-openshift-default Scaled down replica set e2e-worker-w2-c2ce75-openshift-default-699569fc6c from 2 to 1 31m Normal Killing pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Stopping container istio-proxy 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-c2ce75-openshift-default New size: 1; reason: All metrics below target 31m Normal SuccessfulDelete replicaset/e2e-worker-w2-c2ce75-openshift-default-699569fc6c Deleted pod: e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g 31m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-2afe58-openshift-default Scaled up replica set e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 from 1 to 2 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 Created pod: e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal Pulled pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Add eth0 [10.132.0.169/23] from ovn-kubernetes 30m Normal Started pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Container started 30m Normal Created pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Container created 30m Normal Killing pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-ttm5d Stopping container istio-proxy 30m Warning ProbeError pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Startup probe error: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Startup probe failed: Get "http://10.132.0.169:15021/healthz/ready": dial tcp 10.132.0.169:15021: connect: connection refused 30m Normal Killing pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-inf-b-f3338e-6c57db899d-q75hd Stopping container payload-processing 30m Warning Unhealthy pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Readiness probe failed: Get "http://10.134.0.152:15021/healthz/ready": read tcp 10.134.0.2:56264->10.134.0.152:15021: read: connection reset by peer 30m Warning ProbeError pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Readiness probe error: Get "http://10.134.0.152:15021/healthz/ready": read tcp 10.134.0.2:56264->10.134.0.152:15021: read: connection reset by peer... 30m Normal Pulled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Started pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Container started 30m Normal Created pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Container created 30m Warning Unhealthy pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Startup probe failed: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Startup probe error: Get "http://10.134.0.152:15021/healthz/ready": dial tcp 10.134.0.152:15021: connect: connection refused... 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-7a71ca-openshift-default Scaled up replica set e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc from 1 to 2 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc Created pod: e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal Killing pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Stopping container payload-processing 29m Normal Pulled pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Container created 29m Normal AddedInterface pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Add eth0 [10.133.1.1/23] from ovn-kubernetes 29m Normal Started pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Container started 29m Warning ProbeError pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Startup probe error: Get "http://10.133.1.1:15021/healthz/ready": dial tcp 10.133.1.1:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Startup probe failed: Get "http://10.133.1.1:15021/healthz/ready": dial tcp 10.133.1.1:15021: connect: connection refused 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning ProbeError pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Readiness probe error: Get "http://10.133.0.254:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 29m Warning Unhealthy pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Readiness probe failed: Get "http://10.133.0.254:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 29m Normal Started pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Container started 29m Normal Pulled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Container created 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-8a5d5e-openshift-default Scaled up replica set e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c from 0 to 1 29m Normal AddedLabel gateway/e2e-shared-b-w0-8a5d5e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-8a5d5e 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Created pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql 29m Normal AddedInterface pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Add eth0 [10.134.0.160/23] from ovn-kubernetes 29m Warning Unhealthy pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Startup probe failed: Get "http://10.133.0.254:15021/healthz/ready": dial tcp 10.133.0.254:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Startup probe error: Get "http://10.133.0.254:15021/healthz/ready": dial tcp 10.133.0.254:15021: connect: connection refused... 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-8a5d5e Scaled up replica set payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f Created pod: payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-8a5d5e Scaled up replica set payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54 Created pod: payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f Error creating: pods "payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e: serviceaccount "payload-processing-e2e-shared-b-w0-8a5d5e" not found 29m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54 Error creating: pods "payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-8a5d5e: serviceaccount "payload-processing-e2e-shared-b-w0-8a5d5e" not found 29m Normal Pulled pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Add eth0 [10.133.1.3/23] from ovn-kubernetes 29m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Add eth0 [10.133.1.2/23] from ovn-kubernetes 29m Normal Started pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Container started 29m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Container created 29m Normal Created pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Container created 29m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Container started 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning Unhealthy pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Readiness probe failed: HTTP probe failed with statuscode: 503 29m Normal Killing pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Stopping container istio-proxy 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning ProbeError pod/e2e-inf-b-f3338e-openshift-default-786cf5c989-f8v4k Readiness probe error: HTTP probe failed with statuscode: 503... 29m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-2b8461-56d688db75-vxr96 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Warning Unhealthy pod/payload-processing-e2e-inf-a-2b8461-86cb99fc6b-tfhpp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 29m Normal Killing pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk Stopping container istio-proxy 29m Normal Killing pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-7s2z7 Stopping container istio-proxy 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-bcd76c-openshift-default Scaled up replica set e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc from 1 to 2 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc Created pod: e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg 29m Normal AddedInterface pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Add eth0 [10.132.0.173/23] from ovn-kubernetes 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Normal Started pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Container started 29m Normal Created pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Container created 29m Normal Pulled pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Warning ProbeError pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Startup probe error: Get "http://10.132.0.173:15021/healthz/ready": dial tcp 10.132.0.173:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Startup probe failed: Get "http://10.132.0.173:15021/healthz/ready": dial tcp 10.132.0.173:15021: connect: connection refused 29m Normal Killing pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Stopping container payload-processing 29m Normal Killing pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Stopping container payload-pre-processing 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled up replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 2 to 3 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Created pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg 29m Normal Created pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Container created 29m Normal AddedInterface pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Add eth0 [10.132.0.179/23] from ovn-kubernetes 29m Normal Pulled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Started pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Container started 29m Warning Unhealthy pod/payload-processing-e2e-inf-b-a5e47c-6cf788968-nrrhj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-a5e47c-c7ddb7f47-7tvj9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 28m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Startup probe failed: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Startup probe error: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused... 28m Normal Killing pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-6wsrl Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Stopping container istio-proxy 28m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 Created pod: e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc 28m Warning Unhealthy pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Readiness probe failed: HTTP probe failed with statuscode: 503 28m Normal AddedLabel gateway/e2e-shared-a-w3-1ba901 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-1ba901 28m Warning ProbeError pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-1ba901-openshift-default Scaled up replica set e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 from 0 to 1 28m Normal AddedInterface pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Add eth0 [10.134.0.163/23] from ovn-kubernetes 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-1ba901-68d8595698 Created pod: payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901: serviceaccount "payload-processing-e2e-shared-a-w3-1ba901" not found 28m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-1ba901-68d8595698 Error creating: pods "payload-processing-e2e-shared-a-w3-1ba901-68d8595698-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-1ba901: serviceaccount "payload-processing-e2e-shared-a-w3-1ba901" not found 28m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-w27kj Stopping container istio-proxy 28m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 4 to 3 28m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 28m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-w27kj 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-1ba901 Scaled up replica set payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5 from 0 to 1 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-1ba901 Scaled up replica set payload-processing-e2e-shared-a-w3-1ba901-68d8595698 from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5 Created pod: payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl 28m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Add eth0 [10.133.1.7/23] from ovn-kubernetes 28m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Container started 28m Normal Started pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Container started 28m Normal Created pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Container created 28m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Container created 28m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Add eth0 [10.133.1.8/23] from ovn-kubernetes 28m Normal Pulled pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-w27kj Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-w27kj Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal AddedInterface pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Add eth0 [10.132.0.182/23] from ovn-kubernetes 28m Normal Created pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Container created 28m Normal Started pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Container started 28m Normal Created pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Container created 28m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Created pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz 28m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Created pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg 28m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Created pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj 28m Normal AddedInterface pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Add eth0 [10.134.0.164/23] from ovn-kubernetes 28m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-8a5d5e-openshift-default Scaled up replica set e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c from 1 to 4 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 28m Normal Pulled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Add eth0 [10.133.1.9/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Started pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Container started 28m Normal Created pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Container created 28m Normal Started pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Container started 28m Normal Pulled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Started pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Container started 28m Normal Created pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Container created 28m Normal Pulled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 Created pod: e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln 28m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-0b2c92-openshift-default Scaled up replica set e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 from 1 to 2 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 28m Normal AddedInterface pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Add eth0 [10.134.0.165/23] from ovn-kubernetes 28m Normal Pulled pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Container created 28m Normal Started pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Container started 28m Warning ProbeError pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Startup probe error: Get "http://10.133.1.9:15021/healthz/ready": dial tcp 10.133.1.9:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Startup probe failed: Get "http://10.134.0.164:15021/healthz/ready": dial tcp 10.134.0.164:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Startup probe failed: Get "http://10.133.1.9:15021/healthz/ready": dial tcp 10.133.1.9:15021: connect: connection refused 28m Warning Unhealthy pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Startup probe failed: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Startup probe error: Get "http://10.134.0.164:15021/healthz/ready": dial tcp 10.134.0.164:15021: connect: connection refused... 28m Warning ProbeError pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Startup probe error: Get "http://10.132.0.182:15021/healthz/ready": dial tcp 10.132.0.182:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Startup probe failed: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Startup probe error: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Startup probe failed: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Startup probe error: Get "http://10.134.0.165:15021/healthz/ready": dial tcp 10.134.0.165:15021: connect: connection refused... 28m Normal ScalingReplicaSet deployment/e2e-ipp-a-76405c-openshift-default Scaled up replica set e2e-ipp-a-76405c-openshift-default-b6b5cbbdf from 0 to 1 28m Normal AddedLabel gateway/e2e-ipp-a-76405c Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-76405c 28m Normal SuccessfulCreate replicaset/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf Created pod: e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 28m Normal AddedInterface pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Add eth0 [10.133.1.10/23] from ovn-kubernetes 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-76405c Scaled up replica set payload-processing-e2e-ipp-a-76405c-6588477f54 from 0 to 1 28m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Add eth0 [10.133.1.12/23] from ovn-kubernetes 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f Error creating: pods "payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-76405c: serviceaccount "payload-processing-e2e-ipp-a-76405c" not found 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f Created pod: payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-76405c Scaled up replica set payload-pre-processing-e2e-ipp-a-76405c-58c594df5f from 0 to 1 28m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-76405c-6588477f54 Error creating: pods "payload-processing-e2e-ipp-a-76405c-6588477f54-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-76405c: serviceaccount "payload-processing-e2e-ipp-a-76405c" not found 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-76405c-6588477f54 Created pod: payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb 27m Normal Pulled pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Add eth0 [10.134.0.168/23] from ovn-kubernetes 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Container created 27m Normal Created pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Container created 27m Normal Started pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Container started 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0e874e-openshift-default Scaled up replica set e2e-shared-b-w5-0e874e-openshift-default-594f895d85 from 2 to 4 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Created pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Created pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws 27m Normal Pulled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default New size: 5; reason: All metrics below target 27m Normal AddedInterface pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Add eth0 [10.134.0.170/23] from ovn-kubernetes 27m Normal Started pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Container started 27m Normal Created pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Container created 27m Normal Pulled pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Add eth0 [10.133.1.13/23] from ovn-kubernetes 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Created pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx 27m Normal Started pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Container started 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-8a5d5e-openshift-default Scaled up replica set e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c from 4 to 5 27m Normal Created pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Container created 27m Normal Started pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Container started 27m Normal Created pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Container created 27m Normal AddedInterface pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Add eth0 [10.132.0.185/23] from ovn-kubernetes 27m Normal Pulled pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-3e8ee4-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf Created pod: e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-3e8ee4-openshift-default Scaled up replica set e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf from 1 to 2 27m Normal AddedInterface pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Add eth0 [10.133.1.14/23] from ovn-kubernetes 27m Normal Created pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Container created 27m Normal Started pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Container started 27m Normal Pulled pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Warning Unhealthy pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Startup probe failed: Get "http://10.134.0.170:15021/healthz/ready": dial tcp 10.134.0.170:15021: connect: connection refused 27m Warning ProbeError pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Startup probe error: Get "http://10.134.0.170:15021/healthz/ready": dial tcp 10.134.0.170:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Startup probe failed: Get "http://10.133.1.13:15021/healthz/ready": dial tcp 10.133.1.13:15021: connect: connection refused 27m Warning Unhealthy pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Startup probe failed: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused 27m Warning ProbeError pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Startup probe error: Get "http://10.133.1.13:15021/healthz/ready": dial tcp 10.133.1.13:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Startup probe error: Get "http://10.132.0.185:15021/healthz/ready": dial tcp 10.132.0.185:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Startup probe failed: Get "http://10.133.1.14:15021/healthz/ready": dial tcp 10.133.1.14:15021: connect: connection refused 27m Warning ProbeError pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Startup probe error: Get "http://10.133.1.14:15021/healthz/ready": dial tcp 10.133.1.14:15021: connect: connection refused... 27m Normal AddedLabel gateway/e2e-ipp-a-a4f385 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-a4f385 27m Normal ScalingReplicaSet deployment/e2e-ipp-a-a4f385-openshift-default Scaled up replica set e2e-ipp-a-a4f385-openshift-default-b44b4b46f from 0 to 1 27m Normal SuccessfulCreate replicaset/e2e-ipp-a-a4f385-openshift-default-b44b4b46f Created pod: e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt 27m Normal AddedInterface pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Add eth0 [10.133.1.15/23] from ovn-kubernetes 27m Normal AddedLabel gateway/e2e-shared-b-w3-03b5b3 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-03b5b3 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-03b5b3-openshift-default Scaled up replica set e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 from 0 to 1 27m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Add eth0 [10.134.0.176/23] from ovn-kubernetes 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-a4f385 Scaled up replica set payload-pre-processing-e2e-ipp-a-a4f385-569996dfff from 0 to 1 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb Error creating: pods "payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-a4f385: serviceaccount "payload-processing-e2e-ipp-a-a4f385" not found 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff Error creating: pods "payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-a4f385: serviceaccount "payload-processing-e2e-ipp-a-a4f385" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff Created pod: payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb Created pod: payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-a4f385 Scaled up replica set payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb from 0 to 1 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Pulled pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Container created 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Container started 27m Normal Started pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Add eth0 [10.133.1.23/23] from ovn-kubernetes 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Add eth0 [10.133.1.24/23] from ovn-kubernetes 27m Normal Created pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Container created 27m Normal Pulled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Container started 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 27m Normal AddedInterface pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Add eth0 [10.132.0.200/23] from ovn-kubernetes 27m Normal Created pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Container created 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 Created pod: e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 Created pod: e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 Created pod: e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc 27m Normal Pulled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Add eth0 [10.132.0.199/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Add eth0 [10.133.1.25/23] from ovn-kubernetes 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-1ba901-openshift-default Scaled up replica set e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 from 1 to 4 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-03b5b3 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c from 0 to 1 27m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74 Error creating: pods "payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3: serviceaccount "payload-processing-e2e-shared-b-w3-03b5b3" not found 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c Error creating: pods "payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3: serviceaccount "payload-processing-e2e-shared-b-w3-03b5b3" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c Created pod: payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p 27m Normal Started pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Container started 27m Normal Created pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Container created 27m Normal Created pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Container created 27m Normal Started pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Container started 27m Normal Pulled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-03b5b3 Scaled up replica set payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74 from 0 to 1 27m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Readiness probe failed: Get "http://10.134.0.163:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74 Created pod: payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn 27m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Add eth0 [10.134.0.177/23] from ovn-kubernetes 27m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Add eth0 [10.134.0.178/23] from ovn-kubernetes 27m Normal Pulled pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Container created 27m Normal Started pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Container started 27m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Container created 27m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Container started 27m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Readiness probe error: Get "http://10.134.0.163:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal Started pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Container started 27m Normal Pulled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Container created 27m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Startup probe failed: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused 27m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Startup probe failed: Get "http://10.132.0.199:15021/healthz/ready": dial tcp 10.132.0.199:15021: connect: connection refused 27m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Startup probe error: Get "http://10.132.0.199:15021/healthz/ready": dial tcp 10.132.0.199:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Startup probe error: Get "http://10.132.0.200:15021/healthz/ready": dial tcp 10.132.0.200:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Startup probe error: Get "http://10.133.1.25:15021/healthz/ready": dial tcp 10.133.1.25:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Startup probe failed: Get "http://10.133.1.25:15021/healthz/ready": dial tcp 10.133.1.25:15021: connect: connection refused 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-96ab73-openshift-default Scaled up replica set e2e-shared-a-w0-96ab73-openshift-default-586f58856c from 1 to 2 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-96ab73-openshift-default-586f58856c Created pod: e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn 27m Normal AddedInterface pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Add eth0 [10.134.0.179/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Startup probe error: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused... 27m Normal Pulled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Container created 27m Normal Started pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Container started 27m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Startup probe failed: Get "http://10.134.0.163:15021/healthz/ready": dial tcp 10.134.0.163:15021: connect: connection refused 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning ProbeError pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Startup probe error: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Startup probe failed: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-51288b-openshift-default Scaled up replica set e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc from 1 to 2 26m Normal AddedInterface pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Add eth0 [10.132.0.201/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc Created pod: e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr 26m Normal Pulled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Container created 26m Normal Started pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Container started 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-51288b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Warning ProbeError pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Startup probe error: Get "http://10.132.0.201:15021/healthz/ready": dial tcp 10.132.0.201:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Startup probe failed: Get "http://10.132.0.201:15021/healthz/ready": dial tcp 10.132.0.201:15021: connect: connection refused 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal ScalingReplicaSet deployment/e2e-ipp-a-3c1b28-openshift-default Scaled up replica set e2e-ipp-a-3c1b28-openshift-default-df56dcb8f from 0 to 1 26m Normal AddedLabel gateway/e2e-ipp-a-3c1b28 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-3c1b28 26m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss 26m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Add eth0 [10.134.0.180/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6 Created pod: payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n 26m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6 Error creating: pods "payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3c1b28: serviceaccount "payload-processing-e2e-ipp-a-3c1b28" not found 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c Error creating: pods "payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3c1b28: serviceaccount "payload-processing-e2e-ipp-a-3c1b28" not found 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-3c1b28 Scaled up replica set payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c from 0 to 1 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c Created pod: payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-3c1b28 Scaled up replica set payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6 from 0 to 1 26m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Add eth0 [10.133.1.29/23] from ovn-kubernetes 26m Normal AddedInterface pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Add eth0 [10.133.1.28/23] from ovn-kubernetes 26m Normal Pulled pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Container created 26m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Created pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Container created 26m Normal Started pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Container started 26m Normal Started pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Container started 26m Warning Unhealthy pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Readiness probe failed: Get "http://10.133.1.10:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning ProbeError pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Readiness probe error: Get "http://10.133.1.10:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Normal Created pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Container created 26m Normal Pulled pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Container started 26m Normal AddedInterface pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Add eth0 [10.134.0.181/23] from ovn-kubernetes 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-1ba901-openshift-default Scaled up replica set e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 from 4 to 5 26m Normal Killing pod/payload-pre-processing-e2e-ipp-a-76405c-58c594df5f-fdm58 Stopping container payload-pre-processing 26m Normal Killing pod/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb Stopping container payload-processing 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w3-1ba901-openshift-default New size: 5; reason: All metrics below target 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965 Created pod: e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz 26m Normal Created pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Container created 26m Normal Started pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Container started 26m Normal Pulled pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning ProbeError pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Startup probe error: Get "http://10.133.1.10:15021/healthz/ready": dial tcp 10.133.1.10:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Startup probe failed: Get "http://10.133.1.10:15021/healthz/ready": dial tcp 10.133.1.10:15021: connect: connection refused 26m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Startup probe error: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Startup probe failed: Get "http://10.134.0.181:15021/healthz/ready": dial tcp 10.134.0.181:15021: connect: connection refused 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-76405c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-76405c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 Created pod: e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-627dfa-openshift-default Scaled up replica set e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 from 1 to 2 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Normal Started pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Container started 26m Normal Created pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Container created 26m Normal Pulled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Add eth0 [10.133.1.30/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Container created 26m Normal Started pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Container started 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-a4f385-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning Unhealthy pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Startup probe failed: Get "http://10.133.1.30:15021/healthz/ready": dial tcp 10.133.1.30:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Startup probe error: Get "http://10.133.1.30:15021/healthz/ready": dial tcp 10.133.1.30:15021: connect: connection refused... 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-a4f385-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning Unhealthy pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Startup probe failed: Get "http://10.133.1.15:15021/healthz/ready": dial tcp 10.133.1.15:15021: connect: connection refused 25m Warning ProbeError pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Startup probe error: Get "http://10.133.1.15:15021/healthz/ready": dial tcp 10.133.1.15:15021: connect: connection refused... 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-76405c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-76405c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-76405c Scaled up replica set payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c from 0 to 1 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-76405c-5496b48fb8 Error creating: pods "payload-processing-e2e-ipp-a-76405c-5496b48fb8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-76405c: serviceaccount "payload-processing-e2e-ipp-a-76405c" not found 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-76405c Scaled up replica set payload-processing-e2e-ipp-a-76405c-5496b48fb8 from 0 to 1 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c Created pod: payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-76405c-5496b48fb8 Created pod: payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c Error creating: pods "payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-76405c: serviceaccount "payload-processing-e2e-ipp-a-76405c" not found 25m Normal AddedInterface pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Add eth0 [10.134.0.183/23] from ovn-kubernetes 25m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 25m Normal Killing pod/payload-processing-e2e-ipp-a-a4f385-7c98f7f7bb-xl7fn Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-e2e-ipp-a-a4f385-569996dfff-zqq8w Stopping container payload-pre-processing 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Add eth0 [10.132.0.207/23] from ovn-kubernetes 25m Normal AddedLabel gateway/e2e-ipp-b-c91909 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-c91909 25m Normal SuccessfulCreate replicaset/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c Created pod: e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz 25m Normal Pulling pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 25m Normal ScalingReplicaSet deployment/e2e-ipp-b-c91909-openshift-default Scaled up replica set e2e-ipp-b-c91909-openshift-default-85ff7d7f8c from 0 to 1 25m Normal Started pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Container started 25m Normal Created pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Container created 25m Normal Pulled pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.926s (1.926s including waiting). Image size: 166612641 bytes. 25m Normal AddedInterface pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Add eth0 [10.133.1.31/23] from ovn-kubernetes 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.482s (1.482s including waiting). Image size: 166612641 bytes. 25m Normal Created pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Container created 25m Normal Started pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Container started 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899 Created pod: payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-c91909 Scaled up replica set payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899 from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-c91909 Scaled up replica set payload-processing-e2e-ipp-b-c91909-6f7df5857b from 0 to 1 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-c91909-6f7df5857b Error creating: pods "payload-processing-e2e-ipp-b-c91909-6f7df5857b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-c91909: serviceaccount "payload-processing-e2e-ipp-b-c91909" not found 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-c91909-6f7df5857b Created pod: payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899 Error creating: pods "payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-c91909: serviceaccount "payload-processing-e2e-ipp-b-c91909" not found 25m Normal Pulled pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal Created pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Container created 25m Normal Created pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Container created 25m Normal Started pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Container started 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Add eth0 [10.133.1.32/23] from ovn-kubernetes 25m Normal AddedInterface pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Add eth0 [10.133.1.33/23] from ovn-kubernetes 25m Normal Started pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Container started 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-a4f385-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-a4f385-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Normal ScalingReplicaSet deployment/e2e-ipp-b-1c8130-openshift-default Scaled up replica set e2e-ipp-b-1c8130-openshift-default-76f8764dd5 from 0 to 1 25m Normal AddedLabel gateway/e2e-ipp-b-1c8130 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-1c8130 25m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj 25m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Add eth0 [10.134.0.184/23] from ovn-kubernetes 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-1c8130 Scaled up replica set payload-processing-e2e-ipp-b-1c8130-66f66dd9f8 from 0 to 1 25m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8 Error creating: pods "payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-1c8130: serviceaccount "payload-processing-e2e-ipp-b-1c8130" not found 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-1c8130 Scaled up replica set payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf from 0 to 1 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf Error creating: pods "payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-1c8130: serviceaccount "payload-processing-e2e-ipp-b-1c8130" not found 25m Normal Created pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Container created 25m Normal Started pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Container started 25m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf Created pod: payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8 Created pod: payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp 25m Normal AddedInterface pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Add eth0 [10.133.1.35/23] from ovn-kubernetes 25m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Add eth0 [10.133.1.34/23] from ovn-kubernetes 25m Normal Started pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Container started 25m Normal Created pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Container created 25m Normal Pulled pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 25m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 25m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Add eth0 [10.132.0.209/23] from ovn-kubernetes 25m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt 25m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs 25m Normal ScalingReplicaSet deployment/e2e-ipp-a-3c1b28-openshift-default Scaled up replica set e2e-ipp-a-3c1b28-openshift-default-df56dcb8f from 1 to 4 25m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Container started 25m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Add eth0 [10.133.1.36/23] from ovn-kubernetes 25m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Container created 25m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Container started 25m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Container created 25m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Readiness probe error: Get "http://10.134.0.180:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Readiness probe failed: Get "http://10.134.0.180:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Add eth0 [10.134.0.185/23] from ovn-kubernetes 25m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Container started 25m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Container created 25m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Startup probe failed: Get "http://10.133.1.36:15021/healthz/ready": dial tcp 10.133.1.36:15021: connect: connection refused 25m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Startup probe error: Get "http://10.132.0.209:15021/healthz/ready": dial tcp 10.132.0.209:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Startup probe failed: Get "http://10.132.0.209:15021/healthz/ready": dial tcp 10.132.0.209:15021: connect: connection refused 25m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Startup probe error: Get "http://10.133.1.36:15021/healthz/ready": dial tcp 10.133.1.36:15021: connect: connection refused... 25m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-96ab73-openshift-default Scaled up replica set e2e-shared-a-w0-96ab73-openshift-default-586f58856c from 2 to 3 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 25m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-96ab73-openshift-default-586f58856c Created pod: e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw 25m Normal AddedInterface pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Add eth0 [10.132.0.210/23] from ovn-kubernetes 25m Normal Pulled pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Created pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Container created 25m Normal Started pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Container started 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-03b5b3-openshift-default Scaled up replica set e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 from 1 to 4 25m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Add eth0 [10.133.1.37/23] from ovn-kubernetes 25m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Startup probe failed: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused 25m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Container started 25m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Container created 25m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Startup probe error: Get "http://10.134.0.185:15021/healthz/ready": dial tcp 10.134.0.185:15021: connect: connection refused... 25m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Container started 25m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Container created 25m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Add eth0 [10.134.0.186/23] from ovn-kubernetes 25m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Add eth0 [10.132.0.211/23] from ovn-kubernetes 25m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Container created 25m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Container started 25m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Container created 25m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Container started 25m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Readiness probe failed: Get "http://10.134.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Readiness probe error: Get "http://10.134.0.176:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning Unhealthy pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Startup probe failed: Get "http://10.132.0.210:15021/healthz/ready": dial tcp 10.132.0.210:15021: connect: connection refused 25m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Container started 25m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Container created 25m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning ProbeError pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Startup probe error: Get "http://10.132.0.210:15021/healthz/ready": dial tcp 10.132.0.210:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Startup probe error: Get "http://10.133.1.37:15021/healthz/ready": dial tcp 10.133.1.37:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Startup probe failed: Get "http://10.133.1.37:15021/healthz/ready": dial tcp 10.133.1.37:15021: connect: connection refused 24m Normal Killing pod/payload-processing-e2e-ipp-a-3c1b28-ffcb7cb6-5fq9n Stopping container payload-processing 24m Normal Killing pod/payload-pre-processing-e2e-ipp-a-3c1b28-5ffddf798c-qvd4k Stopping container payload-pre-processing 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5 Created pod: payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Startup probe failed: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-a4f385 Scaled up replica set payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5 from 0 to 1 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5 Error creating: pods "payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-a4f385: serviceaccount "payload-processing-e2e-ipp-a-a4f385" not found 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-a4f385-587868497b Error creating: pods "payload-processing-e2e-ipp-a-a4f385-587868497b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-a4f385: serviceaccount "payload-processing-e2e-ipp-a-a4f385" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-a4f385 Scaled up replica set payload-processing-e2e-ipp-a-a4f385-587868497b from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-a4f385-587868497b Created pod: payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Startup probe error: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Startup probe error: Get "http://10.134.0.180:15021/healthz/ready": dial tcp 10.134.0.180:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Startup probe failed: Get "http://10.134.0.180:15021/healthz/ready": dial tcp 10.134.0.180:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Startup probe error: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Startup probe failed: Get "http://10.134.0.186:15021/healthz/ready": dial tcp 10.134.0.186:15021: connect: connection refused 24m Normal Created pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Container created 24m Normal Started pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Container started 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Add eth0 [10.134.0.187/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Container created 24m Normal Started pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Container started 24m Normal Pulled pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Add eth0 [10.134.0.188/23] from ovn-kubernetes 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Startup probe failed: Get "http://10.134.0.176:15021/healthz/ready": dial tcp 10.134.0.176:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Startup probe error: Get "http://10.134.0.176:15021/healthz/ready": dial tcp 10.134.0.176:15021: connect: connection refused... 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-3c1b28-openshift-default Scaled up replica set e2e-ipp-a-3c1b28-openshift-default-df56dcb8f from 4 to 8 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Add eth0 [10.132.0.215/23] from ovn-kubernetes 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Container created 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Container started 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Add eth0 [10.133.1.39/23] from ovn-kubernetes 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Add eth0 [10.133.1.40/23] from ovn-kubernetes 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Container created 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Container created 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Container created 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Container started 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Container started 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Container started 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Add eth0 [10.134.0.189/23] from ovn-kubernetes 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Startup probe error: Get "http://10.134.0.189:15021/healthz/ready": dial tcp 10.134.0.189:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Startup probe failed: Get "http://10.134.0.189:15021/healthz/ready": dial tcp 10.134.0.189:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Startup probe error: Get "http://10.132.0.215:15021/healthz/ready": dial tcp 10.132.0.215:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Startup probe error: Get "http://10.133.1.40:15021/healthz/ready": dial tcp 10.133.1.40:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Startup probe failed: Get "http://10.132.0.215:15021/healthz/ready": dial tcp 10.132.0.215:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Startup probe failed: Get "http://10.133.1.40:15021/healthz/ready": dial tcp 10.133.1.40:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Startup probe error: Get "http://10.133.1.39:15021/healthz/ready": dial tcp 10.133.1.39:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Startup probe failed: Get "http://10.133.1.39:15021/healthz/ready": dial tcp 10.133.1.39:15021: connect: connection refused 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g 24m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-03b5b3-openshift-default Scaled up replica set e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 from 4 to 8 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default New size: 8; reason: All metrics below target 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Container started 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Container created 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Container created 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Add eth0 [10.133.1.41/23] from ovn-kubernetes 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Container started 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Container created 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Add eth0 [10.132.0.219/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Add eth0 [10.132.0.218/23] from ovn-kubernetes 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Add eth0 [10.133.1.42/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Container created 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Container started 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Container started 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-3c1b28-openshift-default Scaled up replica set e2e-ipp-a-3c1b28-openshift-default-df56dcb8f from 8 to 10 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default New size: 10; reason: 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f (combined from similar events): Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Created pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Add eth0 [10.132.0.220/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Add eth0 [10.133.1.43/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc Created pod: e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Container started 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Container created 24m Normal Pulled pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Container created 24m Normal Started pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Container started 24m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-51288b-openshift-default Scaled up replica set e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc from 2 to 3 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-51288b-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Startup probe error: Get "http://10.132.0.219:15021/healthz/ready": dial tcp 10.132.0.219:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Startup probe failed: Get "http://10.132.0.218:15021/healthz/ready": dial tcp 10.132.0.218:15021: connect: connection refused 24m Normal Started pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Container started 24m Normal Created pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Container created 24m Normal Pulled pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Add eth0 [10.133.1.44/23] from ovn-kubernetes 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Startup probe error: Get "http://10.133.1.41:15021/healthz/ready": dial tcp 10.133.1.41:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Startup probe error: Get "http://10.132.0.218:15021/healthz/ready": dial tcp 10.132.0.218:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Startup probe failed: Get "http://10.132.0.219:15021/healthz/ready": dial tcp 10.132.0.219:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Startup probe failed: Get "http://10.133.1.42:15021/healthz/ready": dial tcp 10.133.1.42:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Startup probe error: Get "http://10.133.1.42:15021/healthz/ready": dial tcp 10.133.1.42:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Startup probe failed: Get "http://10.133.1.41:15021/healthz/ready": dial tcp 10.133.1.41:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Startup probe error: Get "http://10.132.0.220:15021/healthz/ready": dial tcp 10.132.0.220:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Startup probe failed: Get "http://10.133.1.43:15021/healthz/ready": dial tcp 10.133.1.43:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Startup probe error: Get "http://10.133.1.43:15021/healthz/ready": dial tcp 10.133.1.43:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Startup probe failed: Get "http://10.132.0.220:15021/healthz/ready": dial tcp 10.132.0.220:15021: connect: connection refused 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-3c1b28-6546f8768 Created pod: payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-3c1b28 Scaled up replica set payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7 from 0 to 1 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default New size: 10; reason: 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-3c1b28-6546f8768 Error creating: pods "payload-processing-e2e-ipp-a-3c1b28-6546f8768-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3c1b28: serviceaccount "payload-processing-e2e-ipp-a-3c1b28" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-3c1b28 Scaled up replica set payload-processing-e2e-ipp-a-3c1b28-6546f8768 from 0 to 1 24m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-03b5b3-openshift-default Scaled up replica set e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 from 8 to 10 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 (combined from similar events): Created pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7 Error creating: pods "payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3c1b28: serviceaccount "payload-processing-e2e-ipp-a-3c1b28" not found 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Add eth0 [10.134.0.194/23] from ovn-kubernetes 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Container created 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Add eth0 [10.134.0.192/23] from ovn-kubernetes 24m Warning ProbeError pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Startup probe error: Get "http://10.133.1.44:15021/healthz/ready": dial tcp 10.133.1.44:15021: connect: connection refused... 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7 Created pod: payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Pulled pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Pulled pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Warning Unhealthy pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Startup probe failed: Get "http://10.133.1.44:15021/healthz/ready": dial tcp 10.133.1.44:15021: connect: connection refused 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Container started 24m Normal AddedInterface pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Add eth0 [10.132.0.221/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Add eth0 [10.134.0.193/23] from ovn-kubernetes 24m Normal Created pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Container created 24m Normal Started pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Container started 24m Normal Started pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Container started 24m Normal Started pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Container started 24m Normal Created pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Container created 24m Normal Created pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Container created 23m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Startup probe failed: Get "http://10.134.0.192:15021/healthz/ready": dial tcp 10.134.0.192:15021: connect: connection refused 23m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Startup probe error: Get "http://10.134.0.192:15021/healthz/ready": dial tcp 10.134.0.192:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Startup probe failed: Get "http://10.132.0.221:15021/healthz/ready": dial tcp 10.132.0.221:15021: connect: connection refused 23m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Startup probe error: Get "http://10.132.0.221:15021/healthz/ready": dial tcp 10.132.0.221:15021: connect: connection refused... 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-72208f-openshift-default-58f8c9f655 Created pod: e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd 23m Normal ScalingReplicaSet deployment/e2e-ipp-b-72208f-openshift-default Scaled up replica set e2e-ipp-b-72208f-openshift-default-58f8c9f655 from 0 to 1 23m Normal AddedLabel gateway/e2e-ipp-b-72208f Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-72208f 23m Normal AddedInterface pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Add eth0 [10.133.1.45/23] from ovn-kubernetes 23m Normal SuccessfulCreate replicaset/e2e-ipp-a-78543b-openshift-default-fb6cd645c Created pod: e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn 23m Normal ScalingReplicaSet deployment/e2e-ipp-a-78543b-openshift-default Scaled up replica set e2e-ipp-a-78543b-openshift-default-fb6cd645c from 0 to 1 23m Normal AddedLabel gateway/e2e-ipp-a-78543b Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-78543b 23m Normal AddedInterface pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Add eth0 [10.132.0.222/23] from ovn-kubernetes 23m Normal Killing pod/payload-pre-processing-e2e-ipp-b-c91909-cfcdc6899-cfn6r Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-ipp-b-c91909-6f7df5857b-k826s Stopping container payload-processing 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-72208f Scaled up replica set payload-pre-processing-e2e-ipp-b-72208f-859dfdb987 from 0 to 1 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-72208f Scaled up replica set payload-processing-e2e-ipp-b-72208f-946784b7b from 0 to 1 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-72208f-946784b7b Error creating: pods "payload-processing-e2e-ipp-b-72208f-946784b7b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-72208f: serviceaccount "payload-processing-e2e-ipp-b-72208f" not found 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987 Created pod: payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl 23m Warning ProbeError pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Readiness probe error: Get "http://10.133.1.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987 Error creating: pods "payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-72208f: serviceaccount "payload-processing-e2e-ipp-b-72208f" not found 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-72208f-946784b7b Created pod: payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 23m Warning Unhealthy pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Readiness probe failed: Get "http://10.133.1.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Pulled pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Container created 23m Normal Created pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Container created 23m Normal Started pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Container started 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Add eth0 [10.133.1.46/23] from ovn-kubernetes 23m Normal AddedInterface pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Add eth0 [10.133.1.47/23] from ovn-kubernetes 23m Normal Started pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Container started 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-78543b Scaled up replica set payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f from 0 to 1 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-78543b Scaled up replica set payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b from 0 to 1 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b Error creating: pods "payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-78543b: serviceaccount "payload-processing-e2e-ipp-a-78543b" not found 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f Created pod: payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b Created pod: payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f Error creating: pods "payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-78543b: serviceaccount "payload-processing-e2e-ipp-a-78543b" not found 23m Normal Created pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Container created 23m Normal Pulled pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Add eth0 [10.134.0.195/23] from ovn-kubernetes 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Add eth0 [10.133.1.48/23] from ovn-kubernetes 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Container created 23m Normal Started pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Container started 23m Normal Started pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Container started 23m Normal Pulled pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Created pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Container created 23m Normal Started pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Container started 23m Normal AddedLabel gateway/e2e-ipp-a-3b47e7 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-3b47e7 23m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-78h7p 23m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Add eth0 [10.134.0.196/23] from ovn-kubernetes 23m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 23m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 23m Normal ScalingReplicaSet deployment/e2e-ipp-a-3b47e7-openshift-default Scaled up replica set e2e-ipp-a-3b47e7-openshift-default-64f995d48c from 0 to 1 23m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w 23m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 23m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Container created 23m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Container started 23m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Created pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g 23m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled up replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 3 to 4 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 23m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Add eth0 [10.132.0.223/23] from ovn-kubernetes 23m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Readiness probe error: HTTP probe failed with statuscode: 503... 23m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Readiness probe failed: HTTP probe failed with statuscode: 503 23m Warning Unhealthy pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Startup probe failed: Get "http://10.133.1.31:15021/healthz/ready": dial tcp 10.133.1.31:15021: connect: connection refused 23m Warning ProbeError pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Startup probe error: Get "http://10.133.1.31:15021/healthz/ready": dial tcp 10.133.1.31:15021: connect: connection refused... 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-c91909-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b Created pod: payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-3b47e7 Scaled up replica set payload-processing-e2e-ipp-a-3b47e7-7598f7d95b from 0 to 1 23m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b Error creating: pods "payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3b47e7: serviceaccount "payload-processing-e2e-ipp-a-3b47e7" not found 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c Created pod: payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-3b47e7 Scaled up replica set payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c from 0 to 1 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-c91909-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c Error creating: pods "payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-3b47e7: serviceaccount "payload-processing-e2e-ipp-a-3b47e7" not found 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 23m Normal ScalingReplicaSet deployment/e2e-ipp-b-1c8130-openshift-default Scaled up replica set e2e-ipp-b-1c8130-openshift-default-76f8764dd5 from 1 to 4 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp 23m Normal Started pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Container started 23m Normal Created pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Container created 23m Normal Pulled pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Add eth0 [10.134.0.198/23] from ovn-kubernetes 23m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Add eth0 [10.134.0.197/23] from ovn-kubernetes 23m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Container created 23m Normal Started pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Container started 23m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Readiness probe error: Get "http://10.134.0.184:15021/healthz/ready": read tcp 10.134.0.2:56174->10.134.0.184:15021: read: connection reset by peer... 23m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Readiness probe failed: Get "http://10.134.0.184:15021/healthz/ready": read tcp 10.134.0.2:56174->10.134.0.184:15021: read: connection reset by peer 23m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Container started 23m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Container created 23m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Killing pod/payload-pre-processing-e2e-ipp-b-1c8130-6db98cd5cf-9982d Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-ipp-b-1c8130-66f66dd9f8-hpcbp Stopping container payload-processing 23m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Startup probe error: Get "http://10.134.0.184:15021/healthz/ready": dial tcp 10.134.0.184:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Startup probe failed: Get "http://10.134.0.184:15021/healthz/ready": dial tcp 10.134.0.184:15021: connect: connection refused 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-c91909-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-c91909-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 23m Normal Pulled pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Add eth0 [10.133.1.49/23] from ovn-kubernetes 23m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 23m Normal Started pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Container started 23m Normal Created pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Container created 23m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Startup probe error: Get "http://10.133.1.49:15021/healthz/ready": dial tcp 10.133.1.49:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Startup probe failed: Get "http://10.133.1.49:15021/healthz/ready": dial tcp 10.133.1.49:15021: connect: connection refused 22m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-c91909 Scaled up replica set payload-processing-e2e-ipp-b-c91909-574f7f7cf8 from 0 to 1 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-c91909 Scaled up replica set payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c from 0 to 1 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c Created pod: payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-c91909-574f7f7cf8 Error creating: pods "payload-processing-e2e-ipp-b-c91909-574f7f7cf8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-c91909: serviceaccount "payload-processing-e2e-ipp-b-c91909" not found 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-c91909-574f7f7cf8 Created pod: payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c Error creating: pods "payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-c91909: serviceaccount "payload-processing-e2e-ipp-b-c91909" not found 22m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 Created pod: e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws 22m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-627dfa-openshift-default Scaled up replica set e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 from 2 to 3 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 22m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z 22m Normal ScalingReplicaSet deployment/e2e-ipp-b-1c8130-openshift-default Scaled up replica set e2e-ipp-b-1c8130-openshift-default-76f8764dd5 from 4 to 7 22m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm 22m Normal SuccessfulCreate replicaset/e2e-ipp-b-1c8130-openshift-default-76f8764dd5 Created pod: e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-1c8130-openshift-default New size: 7; reason: All metrics below target 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-1c8130 Scaled up replica set payload-processing-e2e-ipp-b-1c8130-f779f66bd from 0 to 1 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-1c8130 Scaled up replica set payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9 from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9 Error creating: pods "payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-1c8130: serviceaccount "payload-processing-e2e-ipp-b-1c8130" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9 Created pod: payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-1c8130-f779f66bd Error creating: pods "payload-processing-e2e-ipp-b-1c8130-f779f66bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-1c8130: serviceaccount "payload-processing-e2e-ipp-b-1c8130" not found 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-1c8130-f779f66bd Created pod: payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn 21m Warning ProbeError pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Readiness probe error: Get "http://10.132.0.222:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Normal AddedLabel gateway/e2e-ipp-a-ba8145 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-ba8145 21m Warning Unhealthy pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Readiness probe failed: Get "http://10.132.0.222:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Normal ScalingReplicaSet deployment/e2e-ipp-a-ba8145-openshift-default Scaled up replica set e2e-ipp-a-ba8145-openshift-default-599d44d7f7 from 0 to 1 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-ba8145-openshift-default-599d44d7f7 Created pod: e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 21m Normal Pulled pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Pulled pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Created pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Container created 21m Normal Started pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Container started 21m Normal Created pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Container created 21m Normal Started pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Container started 21m Warning Unhealthy pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Startup probe failed: Get "http://10.133.1.45:15021/healthz/ready": dial tcp 10.133.1.45:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Startup probe error: Get "http://10.133.1.45:15021/healthz/ready": dial tcp 10.133.1.45:15021: connect: connection refused... 21m Warning ProbeError pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Startup probe error: Get "http://10.132.0.222:15021/healthz/ready": dial tcp 10.132.0.222:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Startup probe failed: Get "http://10.132.0.222:15021/healthz/ready": dial tcp 10.132.0.222:15021: connect: connection refused 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 21m Normal ScalingReplicaSet deployment/e2e-ipp-a-3b47e7-openshift-default Scaled up replica set e2e-ipp-a-3b47e7-openshift-default-64f995d48c from 1 to 4 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-78543b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-78543b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Readiness probe failed: Get "http://10.134.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Readiness probe error: Get "http://10.134.0.196:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Container created 21m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Container started 21m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Killing pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Stopping container payload-pre-processing 21m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Startup probe failed: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused 21m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Startup probe error: Get "http://10.134.0.196:15021/healthz/ready": dial tcp 10.134.0.196:15021: connect: connection refused... 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-72208f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-72208f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-78543b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-78543b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-72208f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-72208f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w3-1ba901-89cbd5cf5-srtbl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-processing-e2e-shared-a-w3-1ba901-68d8595698-gjgpk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Normal Killing pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd Stopping container istio-proxy 21m Normal SuccessfulDelete replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Deleted pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-bgtvd 21m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0e874e-openshift-default Scaled down replica set e2e-shared-b-w5-0e874e-openshift-default-594f895d85 from 4 to 3 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default New size: 3; reason: All metrics below target 21m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Add eth0 [10.133.1.55/23] from ovn-kubernetes 21m Normal Pulling pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 21m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.761s (1.761s including waiting). Image size: 166612641 bytes. 21m Normal Created pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Container created 21m Normal Started pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Container started 21m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Add eth0 [10.133.1.56/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Container started 21m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Container created 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default New size: 8; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-ipp-a-3b47e7-openshift-default Scaled up replica set e2e-ipp-a-3b47e7-openshift-default-64f995d48c from 4 to 8 21m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Startup probe error: Get "http://10.133.1.56:15021/healthz/ready": dial tcp 10.133.1.56:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Startup probe failed: Get "http://10.133.1.56:15021/healthz/ready": dial tcp 10.133.1.56:15021: connect: connection refused 21m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Add eth0 [10.133.1.57/23] from ovn-kubernetes 21m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Container created 21m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Container started 20m Normal Killing pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Stopping container istio-proxy 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Startup probe error: Get "http://10.133.1.57:15021/healthz/ready": dial tcp 10.133.1.57:15021: connect: connection refused... 20m Normal Killing pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Stopping container istio-proxy 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Startup probe failed: Get "http://10.133.1.57:15021/healthz/ready": dial tcp 10.133.1.57:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-kdfsl Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-4vcvc Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-stb4h Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning Unhealthy pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc Readiness probe error: HTTP probe failed with statuscode: 503... 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Add eth0 [10.132.0.224/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Add eth0 [10.133.1.59/23] from ovn-kubernetes 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Add eth0 [10.134.0.200/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Container created 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Container started 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Add eth0 [10.134.0.199/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Container created 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Container created 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Add eth0 [10.132.0.225/23] from ovn-kubernetes 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Container created 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Container started 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Container started 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Container created 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Container started 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Container started 20m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k 20m Normal ScalingReplicaSet deployment/e2e-ipp-a-3b47e7-openshift-default Scaled up replica set e2e-ipp-a-3b47e7-openshift-default-64f995d48c from 8 to 10 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3b47e7-openshift-default New size: 10; reason: 20m Normal SuccessfulCreate replicaset/e2e-ipp-a-3b47e7-openshift-default-64f995d48c (combined from similar events): Created pod: e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww 20m Normal Killing pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Stopping container payload-pre-processing 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Startup probe failed: Get "http://10.133.1.59:15021/healthz/ready": dial tcp 10.133.1.59:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Startup probe error: Get "http://10.133.1.59:15021/healthz/ready": dial tcp 10.133.1.59:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Startup probe error: Get "http://10.134.0.200:15021/healthz/ready": dial tcp 10.134.0.200:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Startup probe failed: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Startup probe failed: Get "http://10.132.0.224:15021/healthz/ready": dial tcp 10.132.0.224:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Startup probe error: Get "http://10.132.0.224:15021/healthz/ready": dial tcp 10.132.0.224:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Startup probe failed: Get "http://10.134.0.200:15021/healthz/ready": dial tcp 10.134.0.200:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Startup probe error: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Startup probe error: Get "http://10.134.0.199:15021/healthz/ready": dial tcp 10.134.0.199:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Startup probe failed: Get "http://10.134.0.199:15021/healthz/ready": dial tcp 10.134.0.199:15021: connect: connection refused 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Add eth0 [10.134.0.201/23] from ovn-kubernetes 20m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal Created pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Container created 20m Normal Started pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Container started 20m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s Stopping container istio-proxy 20m Normal SuccessfulDelete replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Deleted pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x 20m Normal SuccessfulDelete replicaset/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f Deleted pod: e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-zbc4s 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-3c1b28-openshift-default New size: 8; reason: All metrics below target 20m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-2xb7x Stopping container istio-proxy 20m Normal ScalingReplicaSet deployment/e2e-ipp-a-3c1b28-openshift-default Scaled down replica set e2e-ipp-a-3c1b28-openshift-default-df56dcb8f from 10 to 8 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Container started 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Add eth0 [10.133.1.63/23] from ovn-kubernetes 20m Normal Started pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Container started 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Container created 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Created pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Container created 20m Normal Pulled pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Add eth0 [10.133.1.64/23] from ovn-kubernetes 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default New size: 2; reason: All metrics below target 20m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-96ab73-openshift-default-586f58856c Deleted pod: e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn 20m Normal Killing pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn Stopping container istio-proxy 20m Normal Killing pod/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl Stopping container payload-processing 20m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-96ab73-openshift-default Scaled down replica set e2e-shared-a-w0-96ab73-openshift-default-586f58856c from 3 to 2 20m Normal Killing pod/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n Stopping container payload-pre-processing 20m Normal AddedInterface pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Add eth0 [10.134.0.202/23] from ovn-kubernetes 20m Normal AddedInterface pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Add eth0 [10.134.0.203/23] from ovn-kubernetes 20m Normal Started pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Container started 20m Normal Created pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Container created 20m Normal Pulled pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal SuccessfulDelete replicaset/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 Deleted pod: e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-03b5b3-openshift-default New size: 9; reason: All metrics below target 20m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-03b5b3-openshift-default Scaled down replica set e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677 from 10 to 9 20m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mztqv Stopping container istio-proxy 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Startup probe error: Get "http://10.133.1.63:15021/healthz/ready": dial tcp 10.133.1.63:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Startup probe failed: Get "http://10.133.1.63:15021/healthz/ready": dial tcp 10.133.1.63:15021: connect: connection refused 20m Normal Created pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Container created 20m Normal Pulled pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Startup probe failed: Get "http://10.133.1.64:15021/healthz/ready": dial tcp 10.133.1.64:15021: connect: connection refused 20m Normal Started pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Container started 20m Normal AddedInterface pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Add eth0 [10.134.0.204/23] from ovn-kubernetes 20m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Startup probe error: Get "http://10.133.1.64:15021/healthz/ready": dial tcp 10.133.1.64:15021: connect: connection refused... 20m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ba8145-564d8b96b6 Created pod: payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z 20m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ba8145 Scaled up replica set payload-processing-e2e-ipp-a-ba8145-564d8b96b6 from 0 to 1 20m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9 Error creating: pods "payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ba8145: serviceaccount "payload-processing-e2e-ipp-a-ba8145" not found 20m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ba8145-564d8b96b6 Error creating: pods "payload-processing-e2e-ipp-a-ba8145-564d8b96b6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ba8145: serviceaccount "payload-processing-e2e-ipp-a-ba8145" not found 20m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9 Created pod: payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j 20m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ba8145 Scaled up replica set payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9 from 0 to 1 19m Normal Started pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Container started 19m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Add eth0 [10.134.0.206/23] from ovn-kubernetes 19m Normal Created pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Container created 19m Normal Pulled pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal Started pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Container started 19m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Add eth0 [10.134.0.205/23] from ovn-kubernetes 19m Normal Created pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Container created 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-5xtt7 Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-s6vxj Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4fnxq Stopping container istio-proxy 19m Normal AddedInterface pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Add eth0 [10.132.0.226/23] from ovn-kubernetes 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-f8pwt Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-4j4ht Stopping container istio-proxy 19m Normal Killing pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Stopping container istio-proxy 19m Normal Created pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Container created 19m Normal Pulled pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Started pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Container started 19m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss Readiness probe error: HTTP probe failed with statuscode: 503... 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Add eth0 [10.132.0.229/23] from ovn-kubernetes 19m Warning ProbeError pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Readiness probe error: Get "http://10.134.0.189:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Add eth0 [10.133.1.68/23] from ovn-kubernetes 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Warning Unhealthy pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj Readiness probe failed: Get "http://10.134.0.189:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Container created 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Container started 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Add eth0 [10.132.0.228/23] from ovn-kubernetes 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Add eth0 [10.132.0.230/23] from ovn-kubernetes 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Container created 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Container started 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Container started 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Container created 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Container started 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Warning Unhealthy pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Startup probe failed: Get "http://10.132.0.226:15021/healthz/ready": dial tcp 10.132.0.226:15021: connect: connection refused 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Container created 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Add eth0 [10.134.0.209/23] from ovn-kubernetes 19m Normal AddedInterface pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Add eth0 [10.134.0.208/23] from ovn-kubernetes 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Container created 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Container started 19m Normal Started pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Container started 19m Warning ProbeError pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Startup probe error: Get "http://10.132.0.226:15021/healthz/ready": dial tcp 10.132.0.226:15021: connect: connection refused... 19m Normal Created pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Container created 19m Normal Pulled pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-qt4l4 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-mkn79 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-stszp Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-f4z7g Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-6rzk7 Stopping container istio-proxy 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Startup probe failed: Get "http://10.133.1.68:15021/healthz/ready": dial tcp 10.133.1.68:15021: connect: connection refused 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Startup probe error: Get "http://10.134.0.209:15021/healthz/ready": dial tcp 10.134.0.209:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Startup probe failed: Get "http://10.134.0.209:15021/healthz/ready": dial tcp 10.134.0.209:15021: connect: connection refused 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Startup probe error: Get "http://10.133.1.68:15021/healthz/ready": dial tcp 10.133.1.68:15021: connect: connection refused... 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Startup probe error: Get "http://10.134.0.208:15021/healthz/ready": dial tcp 10.134.0.208:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Startup probe failed: Get "http://10.134.0.208:15021/healthz/ready": dial tcp 10.134.0.208:15021: connect: connection refused 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Startup probe failed: Get "http://10.132.0.229:15021/healthz/ready": dial tcp 10.132.0.229:15021: connect: connection refused 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Startup probe failed: Get "http://10.132.0.230:15021/healthz/ready": dial tcp 10.132.0.230:15021: connect: connection refused 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Startup probe error: Get "http://10.132.0.228:15021/healthz/ready": dial tcp 10.132.0.228:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Startup probe failed: Get "http://10.132.0.228:15021/healthz/ready": dial tcp 10.132.0.228:15021: connect: connection refused 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Startup probe error: Get "http://10.132.0.230:15021/healthz/ready": dial tcp 10.132.0.230:15021: connect: connection refused... 19m Warning ProbeError pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Startup probe error: Get "http://10.132.0.229:15021/healthz/ready": dial tcp 10.132.0.229:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Readiness probe failed: Get "http://10.133.1.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 19m Warning ProbeError pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-zwxb2 Readiness probe error: Get "http://10.133.1.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 19m Normal Killing pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Stopping container payload-processing 19m Normal Killing pod/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj Stopping container payload-pre-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-09db16-b446f45f9-bb7l4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5 Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p Stopping container payload-processing 19m Normal Killing pod/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-ipp-a-76405c-74df6f7f6c-zrkng Stopping container payload-pre-processing 18m Normal Killing pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Stopping container istio-proxy 18m Warning ProbeError pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-ipp-a-a4f385-openshift-default-b44b4b46f-4ncpt Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning Unhealthy pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Stopping container istio-proxy 18m Warning ProbeError pod/e2e-ipp-a-76405c-openshift-default-b6b5cbbdf-hhpc7 Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal Killing pod/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8 Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-78543b-8b66cb5f-8vftg Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-b-72208f-946784b7b-ww7z4 Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-ipp-b-72208f-859dfdb987-89xnl Stopping container payload-pre-processing 18m Normal Killing pod/e2e-ipp-a-78543b-openshift-default-fb6cd645c-j29dn Stopping container istio-proxy 18m Normal Killing pod/payload-pre-processing-e2e-ipp-b-1c8130-944b84ff9-t2fck Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn Stopping container payload-processing 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w Stopping container istio-proxy 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default New size: 3; reason: All metrics below target 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled down replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 4 to 3 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-bcd76c-openshift-default Scaled down replica set e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc from 2 to 1 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-bcd76c-openshift-default New size: 1; reason: All metrics below target 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-x5wvs Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-d9p4j Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0e874e-openshift-default Scaled down replica set e2e-shared-b-w5-0e874e-openshift-default-594f895d85 from 3 to 2 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w5-0e874e-openshift-default-594f895d85 Deleted pod: e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0e874e-openshift-default New size: 2; reason: All metrics below target 18m Normal SuccessfulDelete replicaset/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc Deleted pod: e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg 18m Normal Killing pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-hmmdg Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Deleted pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g 18m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb Stopping container payload-processing 18m Normal Killing pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wvl7g Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws Stopping container istio-proxy 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-jf2ww Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-gw28k Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-49p5z Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf Deleted pod: e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-3e8ee4-openshift-default Scaled down replica set e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf from 2 to 1 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-qb2xf Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-plv4r Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-cxbtm Stopping container istio-proxy 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-3e8ee4-openshift-default New size: 1; reason: All metrics below target 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-z9p6p Stopping container istio-proxy 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-7qwbn Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-7a71ca-openshift-default New size: 1; reason: All metrics below target 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-dx8fj Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-7a71ca-openshift-default Scaled down replica set e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc from 2 to 1 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm Stopping container istio-proxy 18m Warning Unhealthy pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Readiness probe failed: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused 18m Warning ProbeError pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-j52bd Readiness probe error: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused... 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc Deleted pod: e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-5wmgm 18m Normal Killing pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 Deleted pod: e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w6-0b2c92-openshift-default New size: 1; reason: All metrics below target 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-0b2c92-openshift-default Scaled down replica set e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47 from 2 to 1 18m Normal ScalingReplicaSet deployment/e2e-worker-w4-67f56e-openshift-default Scaled down replica set e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 from 2 to 1 18m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7 Deleted pod: e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-67f56e-openshift-default New size: 1; reason: All metrics below target 18m Normal Started pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Container started 18m Normal Created pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Container created 18m Normal Pulled pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Warning ProbeError pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-kccft Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Stopping container payload-pre-processing 18m Normal Killing pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-2afe58-openshift-default Scaled down replica set e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 from 2 to 1 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default New size: 1; reason: All metrics below target 18m Normal SuccessfulDelete replicaset/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787 Deleted pod: e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-rslmn 18m Normal Killing pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Stopping container payload-processing 18m Warning Unhealthy pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Startup probe failed: Get "http://10.134.0.202:15021/healthz/ready": dial tcp 10.134.0.202:15021: connect: connection refused 18m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Stopping container payload-pre-processing 18m Warning ProbeError pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Startup probe error: Get "http://10.134.0.202:15021/healthz/ready": dial tcp 10.134.0.202:15021: connect: connection refused... 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-8a5d5e-openshift-default New size: 3; reason: All metrics below target 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-8a5d5e-openshift-default Scaled down replica set e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c from 5 to 3 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Deleted pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj 18m Normal Killing pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c Deleted pod: e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z Stopping container payload-processing 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-ba8145-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal Killing pod/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js Stopping container payload-processing 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-ba8145-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal Killing pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-nfbzx Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-58c6585447 from 3 to 2 18m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 18m Normal Killing pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Stopping container istio-proxy 18m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-58c6585447 Deleted pod: maas-default-gateway-openshift-default-58c6585447-5tgdl 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-5tgdl Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-96ab73-openshift-default New size: 1; reason: All metrics below target 18m Normal SuccessfulDelete replicaset/e2e-shared-a-w0-96ab73-openshift-default-586f58856c Deleted pod: e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-96ab73-openshift-default Scaled down replica set e2e-shared-a-w0-96ab73-openshift-default-586f58856c from 2 to 1 18m Normal Killing pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-bnlgw Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Stopping container istio-proxy 18m Warning FailedUpdateStatus horizontalpodautoscaler/e2e-shared-a-w1-2afe58-openshift-default horizontalpodautoscalers.autoscaling "e2e-shared-a-w1-2afe58-openshift-default" not found 18m Normal Killing pod/e2e-shared-a-w1-2afe58-openshift-default-644cdb5787-bv82q Stopping container istio-proxy 18m Warning Unhealthy pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning Unhealthy pod/payload-processing-e2e-shared-a-w1-2afe58-6dd65b9876-gnf2h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-processing-e2e-shared-a-w2-acd125-565744995b-8p7qx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning ProbeError pod/e2e-ipp-b-c91909-openshift-default-85ff7d7f8c-k5rzz Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal Killing pod/e2e-ipp-b-72208f-openshift-default-58f8c9f655-9w5fd Stopping container istio-proxy 18m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w2-acd125-58755586f5-sdjbq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w1-2afe58-5cf84c599-9v52m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Killing pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Stopping container istio-proxy 17m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-ba8145-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning Unhealthy pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-ba8145-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning ProbeError pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 Readiness probe error: HTTP probe failed with statuscode: 503... 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Stopping container payload-pre-processing 17m Normal Killing pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Stopping container istio-proxy 17m Normal SuccessfulDelete replicaset/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc Deleted pod: e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr 17m Normal Killing pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-5vgdr Stopping container istio-proxy 17m Warning Unhealthy pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-6qnmt Readiness probe error: HTTP probe failed with statuscode: 503... 17m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-51288b-openshift-default New size: 2; reason: All metrics below target 17m Normal Killing pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Stopping container payload-processing 17m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-51288b-openshift-default Scaled down replica set e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc from 3 to 2 17m Normal Killing pod/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n Stopping container payload-processing 17m Normal Killing pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6 Stopping container payload-pre-processing 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Stopping container payload-pre-processing 17m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-627dfa-openshift-default New size: 2; reason: All metrics below target 17m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-627dfa-openshift-default Scaled down replica set e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 from 3 to 2 17m Normal Killing pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws Stopping container istio-proxy 17m Normal SuccessfulDelete replicaset/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85 Deleted pod: e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-b79ws 17m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Stopping container payload-processing 17m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-e9d641-openshift-default New size: 2; reason: All metrics below target 17m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-e9d641-openshift-default Scaled down replica set e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 from 3 to 2 17m Normal Killing pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Stopping container istio-proxy 17m Normal Killing pod/payload-processing-e2e-shared-b-w0-8a5d5e-6df7c77c54-kb5ss Stopping container payload-processing 17m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7 Deleted pod: e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Stopping container payload-pre-processing 17m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t Readiness probe failed: HTTP probe failed with statuscode: 503 17m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ba8145 Scaled up replica set payload-processing-e2e-ipp-a-ba8145-5d7fc5698d from 0 to 1 17m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d Error creating: pods "payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ba8145: serviceaccount "payload-processing-e2e-ipp-a-ba8145" not found 17m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7 Created pod: payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 17m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7 Error creating: pods "payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ba8145: serviceaccount "payload-processing-e2e-ipp-a-ba8145" not found 17m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ba8145 Scaled up replica set payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7 from 0 to 1 17m Normal Started pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Container started 17m Warning Unhealthy pod/payload-processing-e2e-shared-b-w2-e9d641-5ccd7cbdcb-57w47 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Started pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Container started 17m Normal Created pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Container created 17m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w2-e9d641-7fc5db5c75-pprkn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Add eth0 [10.134.0.212/23] from ovn-kubernetes 17m Normal Created pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Container created 17m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d Created pod: payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 17m Normal Pulled pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Add eth0 [10.134.0.211/23] from ovn-kubernetes 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w0-8a5d5e-6b8964b67f-lq565 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w5-bcd76c-84f99b655-gnmf7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Killing pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Stopping container istio-proxy 17m Warning Unhealthy pod/payload-processing-e2e-shared-a-w5-bcd76c-7b78578448-ftw8p Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-processing-e2e-shared-b-w1-7a71ca-845d74bc48-j6pcb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Killing pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Stopping container istio-proxy 17m Normal Killing pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-wj5vj Stopping container istio-proxy 17m Warning Unhealthy pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning Unhealthy pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-mqpjg Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning ProbeError pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-fxkhg Readiness probe error: HTTP probe failed with statuscode: 503... 17m Normal Killing pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 Stopping container istio-proxy 17m Normal ScalingReplicaSet deployment/e2e-ipp-b-87e742-openshift-default Scaled up replica set e2e-ipp-b-87e742-openshift-default-8bc9c5b8c from 0 to 1 17m Normal Killing pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-cz9sz Stopping container istio-proxy 17m Normal AddedLabel gateway/e2e-ipp-b-87e742 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-87e742 17m Normal SuccessfulCreate replicaset/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c Created pod: e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r 17m Normal Killing pod/e2e-shared-a-w5-bcd76c-openshift-default-5d97fd74dc-grlrs Stopping container istio-proxy 17m Normal Killing pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-njzmx Stopping container istio-proxy 17m Normal Killing pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-f8skg Stopping container istio-proxy 17m Normal AddedInterface pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Add eth0 [10.133.1.76/23] from ovn-kubernetes 17m Normal Killing pod/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8 Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Stopping container payload-pre-processing 17m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-87e742-77fb4fff4d Error creating: pods "payload-processing-e2e-ipp-b-87e742-77fb4fff4d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-87e742: serviceaccount "payload-processing-e2e-ipp-b-87e742" not found 17m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-87e742-77fb4fff4d Created pod: payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 17m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-87e742 Scaled up replica set payload-processing-e2e-ipp-b-87e742-77fb4fff4d from 0 to 1 17m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c Error creating: pods "payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-87e742: serviceaccount "payload-processing-e2e-ipp-b-87e742" not found 17m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c Created pod: payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 17m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-87e742 Scaled up replica set payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c from 0 to 1 17m Normal Started pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Container started 17m Normal Started pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Container started 17m Normal Created pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Container created 17m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 17m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Add eth0 [10.133.1.78/23] from ovn-kubernetes 17m Normal AddedInterface pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Add eth0 [10.134.0.213/23] from ovn-kubernetes 17m Normal Pulled pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 17m Normal Created pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Container created 17m Normal Killing pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Stopping container payload-pre-processing 17m Normal Killing pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Stopping container payload-processing 17m Normal Killing pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Stopping container payload-pre-processing 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-c2ce75-55d76fd4b7-sc9s5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-processing-e2e-worker-w2-c2ce75-54bfb4c58d-4pmgw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-b-w6-3e8ee4-openshift-default-66597988bf-plfmd Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Stopping container istio-proxy 16m Warning Unhealthy pod/payload-processing-e2e-worker-w1-8b46c9-66fb54bdc6-7b6k9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-djqrv Stopping container istio-proxy 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-8b46c9-c88dbd8f5-jdm7j Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-5pt5h Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-c90b9e-c5c9bcff-fgp9p Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-processing-e2e-worker-w0-c90b9e-5d546d5c54-cgxx9 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Stopping container payload-pre-processing 16m Normal Killing pod/payload-pre-processing-e2e-worker-w5-7a3603-5fb5684f75-bsmrj Stopping container payload-pre-processing 16m Warning Unhealthy pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Created pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Container created 16m Normal Started pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Container started 16m Warning ProbeError pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Readiness probe error: HTTP probe failed with statuscode: 503... 16m Normal Killing pod/payload-processing-e2e-worker-w5-7a3603-6cc7dfb7dd-2lrtt Stopping container payload-processing 16m Normal Killing pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-sflh9 Stopping container istio-proxy 16m Normal Pulled pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 16m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh Stopping container istio-proxy 16m Normal Killing pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-kbw5h Stopping container istio-proxy 16m Normal Killing pod/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8 Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-ipp-b-87e742-69d7859b6c-zt6j6 Stopping container payload-pre-processing 16m Warning Unhealthy pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Startup probe failed: Get "http://10.133.1.76:15021/healthz/ready": dial tcp 10.133.1.76:15021: connect: connection refused 16m Warning ProbeError pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Startup probe error: Get "http://10.133.1.76:15021/healthz/ready": dial tcp 10.133.1.76:15021: connect: connection refused... 16m Normal Killing pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-mbsw8 Stopping container istio-proxy 16m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-87e742 Scaled up replica set payload-processing-e2e-ipp-b-87e742-f67665954 from 0 to 1 16m Warning Unhealthy pod/payload-processing-e2e-worker-w6-dd90cb-54889f9c66-9x5rg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75 Created pod: payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 16m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-87e742-f67665954 Error creating: pods "payload-processing-e2e-ipp-b-87e742-f67665954-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-87e742: serviceaccount "payload-processing-e2e-ipp-b-87e742" not found 16m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-87e742-f67665954 Created pod: payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 16m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75 Error creating: pods "payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-87e742: serviceaccount "payload-processing-e2e-ipp-b-87e742" not found 16m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-87e742 Scaled up replica set payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75 from 0 to 1 16m Normal Created pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Container created 16m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Add eth0 [10.134.0.214/23] from ovn-kubernetes 16m Normal Pulled pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 16m Normal Created pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Container created 16m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 16m Normal AddedInterface pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Add eth0 [10.133.1.84/23] from ovn-kubernetes 16m Normal Started pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Container started 16m Normal Started pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Container started 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-dd90cb-675cdc7bff-f7nwb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-87e742-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-87e742-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl Stopping container istio-proxy 16m Normal Killing pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-njp6b Stopping container istio-proxy 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-87e742-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 16m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-87e742-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-wsjxt Stopping container istio-proxy 15m Normal Killing pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Stopping container istio-proxy 15m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2 Stopping container payload-pre-processing 15m Normal Killing pod/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786 Stopping container payload-processing 15m Warning ProbeError pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 Readiness probe failed: HTTP probe failed with statuscode: 503 15m Normal Killing pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Stopping container istio-proxy 15m Warning ProbeError pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 Readiness probe failed: HTTP probe failed with statuscode: 503 15m Normal Killing pod/payload-processing-e2e-ipp-b-87e742-f67665954-qqm68 Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2 Stopping container payload-pre-processing 15m Normal Killing pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Stopping container istio-proxy 15m Warning Unhealthy pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-ipp-b-87e742-openshift-default-8bc9c5b8c-75q8r Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Stopping container payload-pre-processing 15m Normal Killing pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-k677z Stopping container istio-proxy 14m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w4-51288b-6d8ddc4667-9pfb6 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-processing-e2e-shared-a-w4-51288b-6b98d6f6d8-scb77 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal Killing pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Stopping container payload-processing 14m Normal Killing pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Stopping container payload-pre-processing 14m Normal Killing pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-xh6q5 Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj Stopping container istio-proxy 14m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-78h7p Stopping container discovery 14m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-78h7p 14m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 14m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 14m Normal Killing pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Stopping container payload-processing 14m Warning Unhealthy pod/payload-processing-e2e-shared-b-w4-627dfa-7c497c8ccd-zfg8q Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal Killing pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Stopping container payload-pre-processing 14m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w4-627dfa-f4f66866c-tg558 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-67f56e-7c6db696c-qpbgj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-processing-e2e-worker-w4-67f56e-56998884fb-jjn78 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal Killing pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-q5rrn Stopping container istio-proxy 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 2 to 3 13m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-ktgn6 13m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Add eth0 [10.134.0.216/23] from ovn-kubernetes 13m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Container created 13m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Container started 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Startup probe failed: Get "http://10.134.0.216:15021/healthz/ready": dial tcp 10.134.0.216:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Startup probe error: Get "http://10.134.0.216:15021/healthz/ready": dial tcp 10.134.0.216:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Startup probe failed: Get "http://10.134.0.216:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 13m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 Startup probe error: Get "http://10.134.0.216:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-m4kcl 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-9cf5j 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-ndplc 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 3 to 6 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-ndplc Add eth0 [10.134.0.217/23] from ovn-kubernetes 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Container started 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Container created 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Add eth0 [10.132.1.0/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-ndplc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-ndplc Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-ndplc Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Add eth0 [10.133.1.95/23] from ovn-kubernetes 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-ndplc Startup probe failed: Get "http://10.134.0.217:15021/healthz/ready": dial tcp 10.134.0.217:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Startup probe failed: Get "http://10.133.1.95:15021/healthz/ready": dial tcp 10.133.1.95:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-9cf5j Startup probe error: Get "http://10.133.1.95:15021/healthz/ready": dial tcp 10.133.1.95:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Startup probe error: Get "http://10.132.1.0:15021/healthz/ready": dial tcp 10.132.1.0:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-m4kcl Startup probe failed: Get "http://10.132.1.0:15021/healthz/ready": dial tcp 10.132.1.0:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-ndplc Startup probe error: Get "http://10.134.0.217:15021/healthz/ready": dial tcp 10.134.0.217:15021: connect: connection refused... 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-4vfv9 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-fdnx5 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-jl8n2 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 6 to 9 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 9; reason: 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Container started 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Add eth0 [10.134.0.218/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Add eth0 [10.133.1.96/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Add eth0 [10.132.1.2/23] from ovn-kubernetes 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Container created 12m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Container created 12m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 12m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-26kf9 12m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 12m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-26kf9 Stopping container discovery 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Startup probe failed: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-4vfv9 Startup probe error: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Startup probe error: Get "http://10.134.0.218:15021/healthz/ready": dial tcp 10.134.0.218:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Startup probe error: Get "http://10.133.1.96:15021/healthz/ready": dial tcp 10.133.1.96:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 Startup probe failed: Get "http://10.134.0.218:15021/healthz/ready": dial tcp 10.134.0.218:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-fdnx5 Startup probe failed: Get "http://10.133.1.96:15021/healthz/ready": dial tcp 10.133.1.96:15021: connect: connection refused 11m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-58c6585447 from 9 to 10 11m Normal Pulled pod/maas-default-gateway-openshift-default-58c6585447-52t9q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 11m Normal AddedInterface pod/maas-default-gateway-openshift-default-58c6585447-52t9q Add eth0 [10.133.1.98/23] from ovn-kubernetes 11m Normal Started pod/maas-default-gateway-openshift-default-58c6585447-52t9q Container started 11m Normal Created pod/maas-default-gateway-openshift-default-58c6585447-52t9q Container created 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-58c6585447 Created pod: maas-default-gateway-openshift-default-58c6585447-52t9q 11m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-52t9q Startup probe error: Get "http://10.133.1.98:15021/healthz/ready": dial tcp 10.133.1.98:15021: connect: connection refused... 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-52t9q Startup probe failed: Get "http://10.133.1.98:15021/healthz/ready": dial tcp 10.133.1.98:15021: connect: connection refused 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-52t9q Startup probe failed: Get "http://10.133.1.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-52t9q Startup probe error: Get "http://10.133.1.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 9m8s Normal AddedLabel gateway/e2e-worker-single-main-f53955 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-f53955 9m8s Normal ScalingReplicaSet deployment/e2e-worker-single-main-f53955-openshift-default Scaled up replica set e2e-worker-single-main-f53955-openshift-default-6c7b767cc7 from 0 to 1 9m8s Normal SuccessfulCreate replicaset/e2e-worker-single-main-f53955-openshift-default-6c7b767cc7 Created pod: e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq 9m7s Normal AddedInterface pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Add eth0 [10.134.0.219/23] from ovn-kubernetes 9m2s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-f53955-7bd7df6546 Created pod: payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 9m2s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-f53955 Scaled up replica set payload-processing-e2e-worker-single-main-f53955-7bd7df6546 from 0 to 1 9m2s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-f53955-9c495b979 Error creating: pods "payload-pre-processing-e2e-worker-single-main-f53955-9c495b979-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f53955: serviceaccount "payload-processing-e2e-worker-single-main-f53955" not found 9m2s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-f53955-9c495b979 Created pod: payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh 9m2s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-f53955-7bd7df6546 Error creating: pods "payload-processing-e2e-worker-single-main-f53955-7bd7df6546-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f53955: serviceaccount "payload-processing-e2e-worker-single-main-f53955" not found 9m2s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-f53955 Scaled up replica set payload-pre-processing-e2e-worker-single-main-f53955-9c495b979 from 0 to 1 9m1s Normal Created pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Container created 9m1s Normal Pulled pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m1s Normal Started pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Container started 9m1s Normal Created pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Container created 9m1s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Add eth0 [10.134.0.221/23] from ovn-kubernetes 9m1s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m1s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Add eth0 [10.134.0.220/23] from ovn-kubernetes 9m1s Normal Started pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Container started 8m49s Warning Unhealthy pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Readiness probe failed: Get "http://10.134.0.219:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 8m49s Warning ProbeError pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Readiness probe error: Get "http://10.134.0.219:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 8m48s Normal Created pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Container created 8m48s Normal Pulled pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m48s Normal Started pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Container started 8m42s Warning ProbeError pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Startup probe error: Get "http://10.134.0.219:15021/healthz/ready": dial tcp 10.134.0.219:15021: connect: connection refused... 8m42s Warning Unhealthy pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Startup probe failed: Get "http://10.134.0.219:15021/healthz/ready": dial tcp 10.134.0.219:15021: connect: connection refused 8m7s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-f53955-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m7s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-f53955-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m1s Normal Killing pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Stopping container payload-processing 8m1s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Stopping container payload-pre-processing 7m52s Normal Killing pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq Stopping container istio-proxy 7m39s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m39s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m52s Warning Unhealthy pod/maas-default-gateway-openshift-default-58c6585447-7grvt Readiness probe failed: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m52s Warning ProbeError pod/maas-default-gateway-openshift-default-58c6585447-7grvt Readiness probe error: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m40s Normal AddedLabel gateway/e2e-worker-single-main-bf43af Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-bf43af 5m39s Normal ScalingReplicaSet deployment/e2e-worker-single-main-bf43af-openshift-default Scaled up replica set e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbf from 0 to 1 5m39s Normal AddedInterface pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Add eth0 [10.134.0.223/23] from ovn-kubernetes 5m39s Normal SuccessfulCreate replicaset/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbf Created pod: e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 5m34s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-bf43af Scaled up replica set payload-processing-e2e-worker-single-main-bf43af-6f55d5df96 from 0 to 1 5m34s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-bf43af Scaled up replica set payload-pre-processing-e2e-worker-single-main-bf43af-799d9f4647 from 0 to 1 5m34s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-bf43af-6f55d5df96 Error creating: pods "payload-processing-e2e-worker-single-main-bf43af-6f55d5df96-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-bf43af: serviceaccount "payload-processing-e2e-worker-single-main-bf43af" not found 5m34s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-bf43af-799d9f4647 Error creating: pods "payload-pre-processing-e2e-worker-single-main-bf43af-799d9f4647-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-bf43af: serviceaccount "payload-processing-e2e-worker-single-main-bf43af" not found 5m33s Normal Pulled pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m33s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Add eth0 [10.134.0.225/23] from ovn-kubernetes 5m33s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-bf43af-799d9f4647 Created pod: payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k 5m33s Normal Started pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Container started 5m33s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m33s Normal Created pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Container created 5m33s Normal Started pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Container started 5m33s Normal Created pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Container created 5m33s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Add eth0 [10.134.0.224/23] from ovn-kubernetes 5m33s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-bf43af-6f55d5df96 Created pod: payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw 5m21s Warning Unhealthy pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Readiness probe failed: Get "http://10.134.0.223:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m21s Warning ProbeError pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Readiness probe error: Get "http://10.134.0.223:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m20s Normal Started pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Container started 5m20s Normal Created pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Container created 5m20s Normal Pulled pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m14s Warning Unhealthy pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Startup probe failed: Get "http://10.134.0.223:15021/healthz/ready": dial tcp 10.134.0.223:15021: connect: connection refused 5m14s Warning ProbeError pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Startup probe error: Get "http://10.134.0.223:15021/healthz/ready": dial tcp 10.134.0.223:15021: connect: connection refused... 4m39s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-bf43af-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m39s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-bf43af-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m37s Normal Killing pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Stopping container payload-processing 4m37s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Stopping container payload-pre-processing 4m27s Normal Killing pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 Stopping container istio-proxy 4m21s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m21s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k Readiness probe failed: service unhealthy (responded with "NOT_SERVING")