=== events in openshift-ingress (2026-10-30T05:54:50Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 22m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 53m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 to ip-10-0-135-98.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn to ip-10-0-134-105.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 to ip-10-0-139-228.ec2.internal 23m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj to ip-10-0-134-105.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 to ip-10-0-134-105.ec2.internal 51m Normal Scheduled pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Successfully assigned openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr to ip-10-0-139-228.ec2.internal 53m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 to ip-10-0-134-105.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 to ip-10-0-134-105.ec2.internal 56m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl to ip-10-0-134-105.ec2.internal 9m8s Normal Scheduled pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Successfully assigned openshift-ingress/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb to ip-10-0-139-228.ec2.internal 50m Normal Scheduled pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Successfully assigned openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz to ip-10-0-135-98.ec2.internal 21m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 to ip-10-0-139-228.ec2.internal 5m35s Normal Scheduled pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Successfully assigned openshift-ingress/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 to ip-10-0-139-228.ec2.internal 52m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 to ip-10-0-135-98.ec2.internal 54m Normal Scheduled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Successfully assigned openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj to ip-10-0-139-228.ec2.internal 17m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr to ip-10-0-139-228.ec2.internal 50m Normal Scheduled pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Successfully assigned openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-qx79q to ip-10-0-134-105.ec2.internal 35m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww to ip-10-0-135-98.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js to ip-10-0-135-98.ec2.internal 16m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf to ip-10-0-139-228.ec2.internal 36m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 to ip-10-0-139-228.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc to ip-10-0-139-228.ec2.internal 34m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc to ip-10-0-134-105.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s to ip-10-0-134-105.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf to ip-10-0-135-98.ec2.internal 35m Normal Scheduled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Successfully assigned openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr to ip-10-0-139-228.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr to ip-10-0-135-98.ec2.internal 54m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g to ip-10-0-134-105.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Successfully assigned openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp to ip-10-0-139-228.ec2.internal 24m Normal Preempted pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Preempted by pod b258ca46-73e4-403f-82f6-74e0cd489856 on node ip-10-0-135-98.ec2.internal 51m Normal Scheduled pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Successfully assigned openshift-ingress/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx to ip-10-0-139-228.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Successfully assigned openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q to ip-10-0-139-228.ec2.internal 33m Normal Scheduled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Successfully assigned openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Successfully assigned openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c to ip-10-0-134-105.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Successfully assigned openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx to ip-10-0-134-105.ec2.internal 24m Warning FailedScheduling pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz to ip-10-0-134-105.ec2.internal 23m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 20m Normal Scheduled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Successfully assigned openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg to ip-10-0-135-98.ec2.internal 46m Normal Scheduled pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Successfully assigned openshift-ingress/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt to ip-10-0-134-105.ec2.internal 23m Normal Scheduled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Successfully assigned openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w to ip-10-0-134-105.ec2.internal 23m Warning FailedScheduling pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Warning FailedScheduling pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 53m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq to ip-10-0-134-105.ec2.internal 32m Normal Scheduled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Successfully assigned openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Successfully assigned openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj to ip-10-0-139-228.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv to ip-10-0-135-98.ec2.internal 20m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk to ip-10-0-139-228.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs to ip-10-0-139-228.ec2.internal 53m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp to ip-10-0-135-98.ec2.internal 49m Normal Scheduled pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Successfully assigned openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm to ip-10-0-135-98.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Successfully assigned openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h to ip-10-0-135-98.ec2.internal 57m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn to ip-10-0-139-228.ec2.internal 53m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l to ip-10-0-134-105.ec2.internal 48m Normal Scheduled pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Successfully assigned openshift-ingress/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs to ip-10-0-135-98.ec2.internal 21m Warning FailedScheduling pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr to ip-10-0-135-98.ec2.internal 29m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 to ip-10-0-135-98.ec2.internal 48m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Successfully assigned openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn to ip-10-0-134-105.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss to ip-10-0-135-98.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk to ip-10-0-134-105.ec2.internal 31m Normal Scheduled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Successfully assigned openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Successfully assigned openshift-ingress/e2e-api-a-543d08-openshift-default-f45f9c965-597sx to ip-10-0-134-105.ec2.internal 23m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 52m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd to ip-10-0-135-98.ec2.internal 24m Normal Preempted pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Preempted by pod b258ca46-73e4-403f-82f6-74e0cd489856 on node ip-10-0-135-98.ec2.internal 55m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 to ip-10-0-135-98.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 53m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q to ip-10-0-135-98.ec2.internal 22m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq to ip-10-0-135-98.ec2.internal 22m Warning FailedScheduling pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Successfully assigned openshift-ingress/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 to ip-10-0-139-228.ec2.internal 23m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Normal Preempted pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Preempted by pod b258ca46-73e4-403f-82f6-74e0cd489856 on node ip-10-0-135-98.ec2.internal 23m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz to ip-10-0-134-105.ec2.internal 71m Normal Scheduled pod/router-default-74b99df667-5m6dw Successfully assigned openshift-ingress/router-default-74b99df667-5m6dw to ip-10-0-139-228.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj to ip-10-0-135-98.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 to ip-10-0-135-98.ec2.internal 71m Warning FailedScheduling pod/router-default-74b99df667-5m6dw no nodes available to schedule pods 77m Warning FailedScheduling pod/router-default-74b99df667-5m6dw no nodes available to schedule pods 52m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx to ip-10-0-139-228.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh to ip-10-0-134-105.ec2.internal 52m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk to ip-10-0-134-105.ec2.internal 45m Normal Scheduled pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Successfully assigned openshift-ingress/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr to ip-10-0-139-228.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp to ip-10-0-135-98.ec2.internal 21m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 to ip-10-0-135-98.ec2.internal 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m to ip-10-0-139-228.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 to ip-10-0-135-98.ec2.internal 55m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv to ip-10-0-139-228.ec2.internal 56m Normal Scheduled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Successfully assigned openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv to ip-10-0-134-105.ec2.internal 24m Normal Preempted pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Preempted by pod b258ca46-73e4-403f-82f6-74e0cd489856 on node ip-10-0-135-98.ec2.internal 45m Normal Scheduled pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Successfully assigned openshift-ingress/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g to ip-10-0-139-228.ec2.internal 24m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt to ip-10-0-135-98.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb to ip-10-0-134-105.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t to ip-10-0-134-105.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv to ip-10-0-134-105.ec2.internal 25m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q to ip-10-0-134-105.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft to ip-10-0-134-105.ec2.internal 41m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc to ip-10-0-139-228.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn to ip-10-0-139-228.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Normal Scheduled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Successfully assigned openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 to ip-10-0-134-105.ec2.internal 46m Normal Scheduled pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Successfully assigned openshift-ingress/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 to ip-10-0-134-105.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 28m Normal Scheduled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Successfully assigned openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Successfully assigned openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg to ip-10-0-139-228.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Successfully assigned openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd to ip-10-0-135-98.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 to ip-10-0-135-98.ec2.internal 39m Normal Scheduled pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Successfully assigned openshift-ingress/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt to ip-10-0-139-228.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 to ip-10-0-139-228.ec2.internal 56m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 to ip-10-0-134-105.ec2.internal 30m Normal Scheduled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Successfully assigned openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v to ip-10-0-139-228.ec2.internal 54m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 to ip-10-0-135-98.ec2.internal 42m Normal Scheduled pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Successfully assigned openshift-ingress/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 to ip-10-0-134-105.ec2.internal 43m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch to ip-10-0-134-105.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Warning FailedScheduling pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw to ip-10-0-135-98.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj to ip-10-0-134-105.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 38m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc to ip-10-0-134-105.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl to ip-10-0-135-98.ec2.internal 40m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg to ip-10-0-139-228.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch to ip-10-0-139-228.ec2.internal 42m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Successfully assigned openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Successfully assigned openshift-ingress/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb to ip-10-0-134-105.ec2.internal 35m Normal Scheduled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Successfully assigned openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 to ip-10-0-135-98.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Successfully assigned openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Successfully assigned openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 to ip-10-0-139-228.ec2.internal 22m Normal Scheduled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Successfully assigned openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 50m Normal Scheduled pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Successfully assigned openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 50m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd to ip-10-0-135-98.ec2.internal 33m Normal Scheduled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Successfully assigned openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Successfully assigned openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q to ip-10-0-135-98.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 42m Normal Scheduled pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk to ip-10-0-134-105.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz to ip-10-0-134-105.ec2.internal 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st to ip-10-0-135-98.ec2.internal 44m Normal Scheduled pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 to ip-10-0-139-228.ec2.internal 45m Normal Scheduled pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr to ip-10-0-134-105.ec2.internal 42m Normal Scheduled pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Successfully assigned openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-scchv to ip-10-0-134-105.ec2.internal 29m Normal Scheduled pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Successfully assigned openshift-ingress/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 44m Normal Scheduled pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh to ip-10-0-139-228.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-52g24 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-52g24 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 58m Normal Scheduled pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k to ip-10-0-135-98.ec2.internal 9m2s Normal Scheduled pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj to ip-10-0-134-105.ec2.internal 5m30s Normal Scheduled pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Successfully assigned openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 to ip-10-0-134-105.ec2.internal 37m Normal Scheduled pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm to ip-10-0-134-105.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv to ip-10-0-134-105.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Successfully assigned openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 to ip-10-0-135-98.ec2.internal 50m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 to ip-10-0-139-228.ec2.internal 20m Normal Scheduled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Successfully assigned openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Successfully assigned openshift-ingress/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd to ip-10-0-139-228.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Successfully assigned openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 to ip-10-0-139-228.ec2.internal 49m Normal Scheduled pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Successfully assigned openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c to ip-10-0-139-228.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf to ip-10-0-135-98.ec2.internal 56m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 to ip-10-0-134-105.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Successfully assigned openshift-ingress/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 to ip-10-0-135-98.ec2.internal 50m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj to ip-10-0-134-105.ec2.internal 42m Normal Scheduled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Successfully assigned openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 to ip-10-0-139-228.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Successfully assigned openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw to ip-10-0-135-98.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 to ip-10-0-135-98.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Successfully assigned openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 to ip-10-0-139-228.ec2.internal 32m Normal Scheduled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Successfully assigned openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz to ip-10-0-134-105.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj to ip-10-0-134-105.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh to ip-10-0-135-98.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd to ip-10-0-139-228.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 to ip-10-0-139-228.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Successfully assigned openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv to ip-10-0-135-98.ec2.internal 34m Normal Scheduled pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q to ip-10-0-134-105.ec2.internal 33m Normal Scheduled pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm to ip-10-0-134-105.ec2.internal 30m Normal Scheduled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Successfully assigned openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l to ip-10-0-134-105.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq to ip-10-0-135-98.ec2.internal 28m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw to ip-10-0-134-105.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft to ip-10-0-135-98.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb to ip-10-0-135-98.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Successfully assigned openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 to ip-10-0-139-228.ec2.internal 56m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Successfully assigned openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb to ip-10-0-135-98.ec2.internal 48m Normal Scheduled pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Successfully assigned openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Successfully assigned openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc to ip-10-0-135-98.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 to ip-10-0-139-228.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 to ip-10-0-134-105.ec2.internal 47m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k to ip-10-0-135-98.ec2.internal 49m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q to ip-10-0-135-98.ec2.internal 28m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Successfully assigned openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w to ip-10-0-135-98.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm to ip-10-0-139-228.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz to ip-10-0-134-105.ec2.internal 65m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j to ip-10-0-135-98.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q to ip-10-0-139-228.ec2.internal 49m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s to ip-10-0-134-105.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr to ip-10-0-135-98.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww to ip-10-0-139-228.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t to ip-10-0-139-228.ec2.internal 34m Normal Scheduled pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Successfully assigned openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 to ip-10-0-134-105.ec2.internal 54m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg to ip-10-0-134-105.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk to ip-10-0-135-98.ec2.internal 44m Normal Scheduled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Successfully assigned openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Successfully assigned openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk to ip-10-0-135-98.ec2.internal 53m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v to ip-10-0-139-228.ec2.internal 56m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl to ip-10-0-135-98.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Successfully assigned openshift-ingress/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 to ip-10-0-139-228.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Successfully assigned openshift-ingress/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Successfully assigned openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j to ip-10-0-134-105.ec2.internal 46m Normal Scheduled pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Successfully assigned openshift-ingress/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 23m Warning FailedScheduling pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 to ip-10-0-139-228.ec2.internal 18m Normal Scheduled pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Successfully assigned openshift-ingress/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 19m Normal Scheduled pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Successfully assigned openshift-ingress/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf to ip-10-0-139-228.ec2.internal 38m Normal Scheduled pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Successfully assigned openshift-ingress/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 65m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-lr8nf Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-lr8nf to ip-10-0-139-228.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 49m Normal Scheduled pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Successfully assigned openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 to ip-10-0-134-105.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Normal Scheduled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Successfully assigned openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b to ip-10-0-135-98.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h to ip-10-0-134-105.ec2.internal 34m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz to ip-10-0-134-105.ec2.internal 39m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Successfully assigned openshift-ingress/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh to ip-10-0-139-228.ec2.internal 47m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k to ip-10-0-135-98.ec2.internal 47m Normal Scheduled pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Successfully assigned openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 to ip-10-0-135-98.ec2.internal 64m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-842vl to ip-10-0-139-228.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Normal Scheduled pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Successfully assigned openshift-ingress/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg to ip-10-0-139-228.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 51m Normal Scheduled pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 to ip-10-0-139-228.ec2.internal 24m Normal Preempted pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Preempted by pod b258ca46-73e4-403f-82f6-74e0cd489856 on node ip-10-0-135-98.ec2.internal 30m Normal Scheduled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Successfully assigned openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm to ip-10-0-135-98.ec2.internal 43m Normal Scheduled pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Successfully assigned openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc to ip-10-0-134-105.ec2.internal 51m Normal Scheduled pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp to ip-10-0-139-228.ec2.internal 24m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb 0/3 nodes are available: 3 Insufficient cpu. 24m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: not eligible due to a terminating pod on the nominated node. 24m Warning FailedScheduling pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: not eligible due to a terminating pod on the nominated node. 24m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb to ip-10-0-135-98.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Successfully assigned openshift-ingress/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q to ip-10-0-139-228.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 to ip-10-0-134-105.ec2.internal 50m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 to ip-10-0-135-98.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm to ip-10-0-134-105.ec2.internal 53m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nxlpl to ip-10-0-134-105.ec2.internal 30m Normal Scheduled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Successfully assigned openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb to ip-10-0-134-105.ec2.internal 48m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk to ip-10-0-134-105.ec2.internal 24m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 to ip-10-0-139-228.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p to ip-10-0-139-228.ec2.internal 24m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-mpg5z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 43m Normal Scheduled pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Successfully assigned openshift-ingress/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs to ip-10-0-139-228.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc to ip-10-0-134-105.ec2.internal 16m Normal Scheduled pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 to ip-10-0-139-228.ec2.internal 17m Normal Scheduled pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l to ip-10-0-139-228.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f to ip-10-0-134-105.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g to ip-10-0-134-105.ec2.internal 21m Normal Scheduled pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 to ip-10-0-134-105.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h to ip-10-0-134-105.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm to ip-10-0-139-228.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-mpg5z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 49m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf to ip-10-0-139-228.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 to ip-10-0-134-105.ec2.internal 21m Normal Scheduled pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Successfully assigned openshift-ingress/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f to ip-10-0-139-228.ec2.internal 31m Normal Scheduled pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr to ip-10-0-139-228.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv to ip-10-0-134-105.ec2.internal 29m Normal Scheduled pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs to ip-10-0-134-105.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq to ip-10-0-134-105.ec2.internal 23m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 21m Normal Scheduled pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w to ip-10-0-135-98.ec2.internal 21m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 22m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 24m Normal Scheduled pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn to ip-10-0-134-105.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Successfully assigned openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 to ip-10-0-134-105.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 to ip-10-0-134-105.ec2.internal 59m Normal Scheduled pod/payload-pre-processing-7b98bb678c-p6s8d Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-p6s8d to ip-10-0-134-105.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Successfully assigned openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x to ip-10-0-139-228.ec2.internal 21m Warning FailedScheduling pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 65m Normal Scheduled pod/kube-auth-proxy-54668d58ff-2xllz Successfully assigned openshift-ingress/kube-auth-proxy-54668d58ff-2xllz to ip-10-0-134-105.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd to ip-10-0-134-105.ec2.internal 33m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs to ip-10-0-139-228.ec2.internal 34m Normal Scheduled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Successfully assigned openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc to ip-10-0-135-98.ec2.internal 41m Normal Scheduled pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf to ip-10-0-134-105.ec2.internal 39m Normal Scheduled pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 to ip-10-0-139-228.ec2.internal 43m Normal Scheduled pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-c7f962-6886679566-84khk to ip-10-0-139-228.ec2.internal 37m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z to ip-10-0-134-105.ec2.internal 5m30s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn to ip-10-0-134-105.ec2.internal 9m2s Normal Scheduled pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx to ip-10-0-134-105.ec2.internal 65m Normal Scheduled pod/kube-auth-proxy-54668d58ff-kgtw4 Successfully assigned openshift-ingress/kube-auth-proxy-54668d58ff-kgtw4 to ip-10-0-139-228.ec2.internal 39m Normal Scheduled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Successfully assigned openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p to ip-10-0-135-98.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 to ip-10-0-135-98.ec2.internal 35m Normal Scheduled pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Successfully assigned openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt to ip-10-0-135-98.ec2.internal 47m Normal Scheduled pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw to ip-10-0-139-228.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq to ip-10-0-134-105.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn to ip-10-0-134-105.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Successfully assigned openshift-ingress/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk to ip-10-0-134-105.ec2.internal 40m Normal Scheduled pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Successfully assigned openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Successfully assigned openshift-ingress/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 to ip-10-0-139-228.ec2.internal 26m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg to ip-10-0-139-228.ec2.internal 50m Normal Scheduled pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Successfully assigned openshift-ingress/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d to ip-10-0-139-228.ec2.internal 42m Normal Scheduled pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Successfully assigned openshift-ingress/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k to ip-10-0-139-228.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z to ip-10-0-134-105.ec2.internal 59m Normal Scheduled pod/payload-processing-5f78687d5b-vqsmz Successfully assigned openshift-ingress/payload-processing-5f78687d5b-vqsmz to ip-10-0-134-105.ec2.internal 41m Normal Scheduled pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Successfully assigned openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn to ip-10-0-134-105.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Successfully assigned openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 to ip-10-0-134-105.ec2.internal 51m Normal Scheduled pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Successfully assigned openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 to ip-10-0-135-98.ec2.internal 33m Normal Scheduled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Successfully assigned openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Successfully assigned openshift-ingress/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm to ip-10-0-139-228.ec2.internal 21m Normal Scheduled pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Successfully assigned openshift-ingress/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w to ip-10-0-134-105.ec2.internal 31m Normal Scheduled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Successfully assigned openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 to ip-10-0-135-98.ec2.internal 36m Normal Scheduled pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Successfully assigned openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw to ip-10-0-135-98.ec2.internal 51m Normal Scheduled pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Successfully assigned openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b to ip-10-0-135-98.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb to ip-10-0-135-98.ec2.internal 55m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs to ip-10-0-139-228.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d to ip-10-0-135-98.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Successfully assigned openshift-ingress/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw to ip-10-0-139-228.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-52stt to ip-10-0-134-105.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn to ip-10-0-139-228.ec2.internal 22m Warning FailedScheduling pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 46m Normal Scheduled pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Successfully assigned openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl to ip-10-0-135-98.ec2.internal 62m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 to ip-10-0-135-98.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 45m Normal Scheduled pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 to ip-10-0-134-105.ec2.internal 22m Warning FailedScheduling pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 55m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 to ip-10-0-135-98.ec2.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x to ip-10-0-135-98.ec2.internal 59m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-672ql to ip-10-0-139-228.ec2.internal 38m Normal Scheduled pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Successfully assigned openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 to ip-10-0-139-228.ec2.internal 46m Normal Scheduled pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Successfully assigned openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 to ip-10-0-135-98.ec2.internal 22m Normal Scheduled pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Successfully assigned openshift-ingress/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq to ip-10-0-134-105.ec2.internal 48m Normal Scheduled pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Successfully assigned openshift-ingress/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk to ip-10-0-134-105.ec2.internal 58m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 to ip-10-0-135-98.ec2.internal 41m Normal Scheduled pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Successfully assigned openshift-ingress/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk to ip-10-0-134-105.ec2.internal 29m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm to ip-10-0-134-105.ec2.internal 46m Normal Scheduled pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Successfully assigned openshift-ingress/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp to ip-10-0-139-228.ec2.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k to ip-10-0-139-228.ec2.internal 43m Normal Scheduled pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Successfully assigned openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 to ip-10-0-134-105.ec2.internal 44m Normal Scheduled pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Successfully assigned openshift-ingress/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp to ip-10-0-139-228.ec2.internal 55m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx to ip-10-0-134-105.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt to ip-10-0-139-228.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-db25s to ip-10-0-134-105.ec2.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx to ip-10-0-139-228.ec2.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 to ip-10-0-134-105.ec2.internal 55m Normal Scheduled pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 to ip-10-0-135-98.ec2.internal 27m Normal Scheduled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Successfully assigned openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n to ip-10-0-139-228.ec2.internal 31m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz to ip-10-0-134-105.ec2.internal 78m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-74b99df667 from 0 to 1 78m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 78m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 77m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 77m Normal SuccessfulCreate replicaset/router-default-74b99df667 Created pod: router-default-74b99df667-5m6dw 71m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 71m Warning FailedMount pod/router-default-74b99df667-5m6dw MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 71m Warning FailedMount pod/router-default-74b99df667-5m6dw MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 70m Warning FailedMount pod/router-default-74b99df667-5m6dw MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 70m Normal Started pod/router-default-74b99df667-5m6dw Container started 70m Normal Created pod/router-default-74b99df667-5m6dw Container created 70m Normal Pulled pod/router-default-74b99df667-5m6dw Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 70m Normal AddedInterface pod/router-default-74b99df667-5m6dw Add eth0 [10.132.0.15/23] from ovn-kubernetes 65m Normal SuccessfulCreate replicaset/kube-auth-proxy-54668d58ff Created pod: kube-auth-proxy-54668d58ff-2xllz 65m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-54668d58ff from 0 to 2 65m Normal SuccessfulCreate replicaset/kube-auth-proxy-54668d58ff Created pod: kube-auth-proxy-54668d58ff-kgtw4 65m Normal AddedInterface pod/kube-auth-proxy-54668d58ff-kgtw4 Add eth0 [10.132.0.34/23] from ovn-kubernetes 65m Normal Pulling pod/kube-auth-proxy-54668d58ff-2xllz Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 65m Normal AddedInterface pod/kube-auth-proxy-54668d58ff-2xllz Add eth0 [10.134.0.20/23] from ovn-kubernetes 65m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 65m Normal Pulling pod/kube-auth-proxy-54668d58ff-kgtw4 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 65m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-lr8nf 65m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-lr8nf Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 65m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-lr8nf Add eth0 [10.132.0.35/23] from ovn-kubernetes 65m Normal Pulled pod/kube-auth-proxy-54668d58ff-2xllz Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.367s (2.367s including waiting). Image size: 188904677 bytes. 65m Normal Created pod/kube-auth-proxy-54668d58ff-2xllz Container created 65m Normal Started pod/kube-auth-proxy-54668d58ff-2xllz Container started 65m Normal Started pod/kube-auth-proxy-54668d58ff-kgtw4 Container started 65m Normal Created pod/kube-auth-proxy-54668d58ff-kgtw4 Container created 65m Normal Pulled pod/kube-auth-proxy-54668d58ff-kgtw4 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.813s (2.813s including waiting). Image size: 188904677 bytes. 65m Normal Created pod/istiod-openshift-gateway-7876bfcb74-lr8nf Container created 65m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-lr8nf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 2.871s (2.871s including waiting). Image size: 185476477 bytes. 65m Normal Started pod/istiod-openshift-gateway-7876bfcb74-lr8nf Container started 65m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-67df64d57 Created pod: data-science-gateway-data-science-gateway-class-67df64d57-nwq4j 65m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-67df64d57 from 0 to 1 65m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Add eth0 [10.133.0.25/23] from ovn-kubernetes 65m Normal Pulling pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 65m Normal Pulled pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.435s (2.435s including waiting). Image size: 183555333 bytes. 65m Normal Created pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Container created 65m Normal Started pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Container started 65m Warning Unhealthy pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Startup probe failed: Get "http://10.133.0.25:15021/healthz/ready": dial tcp 10.133.0.25:15021: connect: connection refused 65m Warning ProbeError pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j Startup probe error: Get "http://10.133.0.25:15021/healthz/ready": dial tcp 10.133.0.25:15021: connect: connection refused... 65m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 65m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 65m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 64m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 64m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 64m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Add eth0 [10.132.0.36/23] from ovn-kubernetes 64m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Container created 64m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 64m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Container started 64m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-842vl 64m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Readiness probe error: HTTP probe failed with statuscode: 503... 64m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Readiness probe failed: HTTP probe failed with statuscode: 503 64m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-lr8nf 64m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 64m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-lr8nf Stopping container discovery 64m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 64m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 62m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Add eth0 [10.133.0.37/23] from ovn-kubernetes 62m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 0 to 1 62m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 62m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 60m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 60m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Container created 60m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Container started 60m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Startup probe failed: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused 60m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Startup probe error: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused... 60m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 60m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 60m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 59m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw 59m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 1 to 2 59m Normal Pulling pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Add eth0 [10.132.0.49/23] from ovn-kubernetes 59m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 2.566s (2.566s including waiting). Image size: 183555333 bytes. 59m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Container started 59m Normal SuccessfulCreate replicaset/payload-processing-5f78687d5b Created pod: payload-processing-5f78687d5b-vqsmz 59m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-p6s8d 59m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5f78687d5b from 0 to 1 59m Warning FailedCreate replicaset/payload-processing-5f78687d5b Error creating: pods "payload-processing-5f78687d5b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 59m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 59m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 59m Normal Pulling pod/payload-processing-5f78687d5b-vqsmz Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 59m Normal AddedInterface pod/payload-processing-5f78687d5b-vqsmz Add eth0 [10.134.0.37/23] from ovn-kubernetes 59m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-p6s8d Add eth0 [10.134.0.38/23] from ovn-kubernetes 59m Normal Pulling pod/payload-pre-processing-7b98bb678c-p6s8d Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 59m Normal Started pod/payload-pre-processing-7b98bb678c-p6s8d Container started 59m Normal Created pod/payload-pre-processing-7b98bb678c-p6s8d Container created 59m Normal Pulled pod/payload-pre-processing-7b98bb678c-p6s8d Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.179s (2.179s including waiting). Image size: 125317101 bytes. 59m Normal Started pod/payload-processing-5f78687d5b-vqsmz Container started 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused... 59m Normal Created pod/payload-processing-5f78687d5b-vqsmz Container created 59m Normal Pulled pod/payload-processing-5f78687d5b-vqsmz Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.213s (2.213s including waiting). Image size: 125317101 bytes. 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": dial tcp 10.132.0.49:15021: connect: connection refused 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Startup probe failed: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Startup probe error: Get "http://10.132.0.49:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 59m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-672ql 59m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 2 to 4 59m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 59m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Container started 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Add eth0 [10.132.0.50/23] from ovn-kubernetes 59m Normal Pulling pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 59m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Container created 59m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Add eth0 [10.134.0.39/23] from ovn-kubernetes 59m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Container created 59m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.375s (3.375s including waiting). Image size: 183555333 bytes. 59m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Container started 59m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Startup probe failed: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused 59m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Startup probe error: Get "http://10.132.0.50:15021/healthz/ready": dial tcp 10.132.0.50:15021: connect: connection refused... 58m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 58m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 4 to 6 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 58m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-52stt 58m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Add eth0 [10.133.0.40/23] from ovn-kubernetes 58m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Add eth0 [10.134.0.41/23] from ovn-kubernetes 58m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 58m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Container created 58m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Container started 58m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Container started 58m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Container created 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 58m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 58m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 58m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 58m Normal AddedLabel gateway/e2e-worker-w3-889ca9 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-889ca9 58m Normal AddedLabel gateway/e2e-worker-w6-83e83e Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-83e83e 58m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 0 to 1 58m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled up replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 0 to 1 58m Normal AddedLabel gateway/e2e-worker-w5-0b21ba Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-0b21ba 58m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j 58m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx 58m Normal AddedLabel gateway/e2e-worker-w1-cff4ac Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-cff4ac 58m Normal AddedLabel gateway/e2e-worker-w2-397179 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-397179 58m Normal AddedLabel gateway/e2e-worker-w4-d55037 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-d55037 58m Normal AddedLabel gateway/e2e-worker-w0-b21507 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-b21507 58m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd 58m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 58m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 0 to 1 58m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb 58m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Add eth0 [10.132.0.52/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c 58m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled up replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 0 to 1 58m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 0 to 1 58m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 58m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 0 to 1 58m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Add eth0 [10.134.0.42/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Add eth0 [10.132.0.51/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Add eth0 [10.132.0.53/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Add eth0 [10.134.0.43/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled up replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 0 to 1 58m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Add eth0 [10.134.0.44/23] from ovn-kubernetes 58m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Add eth0 [10.132.0.54/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-397179 Scaled up replica set payload-processing-e2e-worker-w2-397179-6db8cdbc74 from 0 to 1 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-397179-6db8cdbc74 Error creating: pods "payload-processing-e2e-worker-w2-397179-6db8cdbc74-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-397179: serviceaccount "payload-processing-e2e-worker-w2-397179" not found 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-397179 Scaled up replica set payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c from 0 to 1 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c Error creating: pods "payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-397179: serviceaccount "payload-processing-e2e-worker-w2-397179" not found 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Add eth0 [10.134.0.45/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c Created pod: payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-397179-6db8cdbc74 Created pod: payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 58m Normal Started pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Container started 58m Normal Started pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Container started 58m Normal Created pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Container created 58m Normal Pulled pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Add eth0 [10.134.0.46/23] from ovn-kubernetes 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Container created 58m Normal Pulled pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Add eth0 [10.134.0.47/23] from ovn-kubernetes 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Add eth0 [10.134.0.48/23] from ovn-kubernetes 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55 Error creating: pods "payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-889ca9: serviceaccount "payload-processing-e2e-worker-w3-889ca9" not found 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55 Created pod: payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-889ca9 Scaled up replica set payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55 from 0 to 1 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-889ca9 Scaled up replica set payload-processing-e2e-worker-w3-889ca9-7dcc755ffb from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb Created pod: payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb Error creating: pods "payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-889ca9: serviceaccount "payload-processing-e2e-worker-w3-889ca9" not found 58m Normal Started pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Container started 58m Normal Created pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Container started 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Container created 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb Created pod: payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5 Error creating: pods "payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-cff4ac: serviceaccount "payload-processing-e2e-worker-w1-cff4ac" not found 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb Error creating: pods "payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-cff4ac: serviceaccount "payload-processing-e2e-worker-w1-cff4ac" not found 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5 Created pod: payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-cff4ac Scaled up replica set payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5 from 0 to 1 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-cff4ac Scaled up replica set payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb from 0 to 1 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-0b21ba Scaled up replica set payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d from 0 to 1 58m Normal AddedInterface pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Add eth0 [10.134.0.49/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7 Created pod: payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k 58m Normal Started pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Container started 58m Normal Created pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Container created 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-0b21ba Scaled up replica set payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7 from 0 to 1 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7 Error creating: pods "payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-0b21ba: serviceaccount "payload-processing-e2e-worker-w5-0b21ba" not found 58m Normal Pulled pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d Error creating: pods "payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-0b21ba: serviceaccount "payload-processing-e2e-worker-w5-0b21ba" not found 58m Normal Started pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Container started 58m Normal Created pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Container created 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Add eth0 [10.134.0.50/23] from ovn-kubernetes 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d Created pod: payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp 58m Normal Started pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Container started 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5 Error creating: pods "payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-b21507: serviceaccount "payload-processing-e2e-worker-w0-b21507" not found 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-b21507-59b47dc888 Error creating: pods "payload-processing-e2e-worker-w0-b21507-59b47dc888-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-b21507: serviceaccount "payload-processing-e2e-worker-w0-b21507" not found 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-b21507 Scaled up replica set payload-processing-e2e-worker-w0-b21507-59b47dc888 from 0 to 1 58m Normal Pulling pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Normal Created pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Container created 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Add eth0 [10.132.0.57/23] from ovn-kubernetes 58m Normal Pulled pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal AddedInterface pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Add eth0 [10.134.0.51/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-b21507 Scaled up replica set payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5 from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5 Created pod: payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Add eth0 [10.133.0.42/23] from ovn-kubernetes 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-d55037 Scaled up replica set payload-processing-e2e-worker-w4-d55037-86f99856b from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-d55037-86f99856b Created pod: payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 58m Normal AddedInterface pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Add eth0 [10.133.0.41/23] from ovn-kubernetes 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-d55037-86f99856b Error creating: pods "payload-processing-e2e-worker-w4-d55037-86f99856b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-d55037: serviceaccount "payload-processing-e2e-worker-w4-d55037" not found 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-d55037 Scaled up replica set payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5 from 0 to 1 58m Normal Pulling pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5 Error creating: pods "payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-d55037: serviceaccount "payload-processing-e2e-worker-w4-d55037" not found 58m Normal Pulling pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5 Created pod: payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-b21507-59b47dc888 Created pod: payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k 58m Normal AddedInterface pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Add eth0 [10.134.0.52/23] from ovn-kubernetes 58m Normal Started pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Container started 58m Normal Pulled pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Container started 58m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Add eth0 [10.134.0.53/23] from ovn-kubernetes 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 58m Normal Created pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Container created 58m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-83e83e Scaled up replica set payload-processing-e2e-worker-w6-83e83e-544bdfd9d from 0 to 1 58m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-83e83e-544bdfd9d Created pod: payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 58m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-83e83e-544bdfd9d Error creating: pods "payload-processing-e2e-worker-w6-83e83e-544bdfd9d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-83e83e: serviceaccount "payload-processing-e2e-worker-w6-83e83e" not found 58m Normal Started pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Container started 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.058s (2.058s including waiting). Image size: 125317101 bytes. 58m Normal Created pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Container created 58m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.99s (2.99s including waiting). Image size: 125317101 bytes. 58m Normal Created pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Container created 58m Normal Started pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Container started 58m Normal Pulled pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.116s (2.116s including waiting). Image size: 125317101 bytes. 58m Normal Created pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Container created 58m Normal Started pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Container started 58m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575 Error creating: pods "payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-83e83e: serviceaccount "payload-processing-e2e-worker-w6-83e83e" not found 58m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575 Created pod: payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z 58m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-83e83e Scaled up replica set payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575 from 0 to 1 57m Normal Created pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Container created 57m Normal Started pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Container started 57m Normal Started pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Container started 57m Normal Created pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Container created 57m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Add eth0 [10.134.0.55/23] from ovn-kubernetes 57m Normal Pulled pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 57m Normal AddedInterface pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Add eth0 [10.134.0.54/23] from ovn-kubernetes 57m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Container started 57m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Container created 57m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Container created 57m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Container started 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Startup probe failed: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Startup probe error: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused... 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Container started 57m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Container created 57m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Readiness probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Readiness probe error: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Readiness probe failed: Get "http://10.134.0.44:15021/healthz/ready": read tcp 10.134.0.2:43158->10.134.0.44:15021: read: connection reset by peer 57m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Container started 57m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Readiness probe error: Get "http://10.134.0.44:15021/healthz/ready": read tcp 10.134.0.2:43158->10.134.0.44:15021: read: connection reset by peer... 57m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Container created 57m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Container started 57m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Container created 57m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Startup probe error: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Startup probe failed: Get "http://10.132.0.54:15021/healthz/ready": dial tcp 10.132.0.54:15021: connect: connection refused 57m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Container created 57m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Container started 57m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Container started 57m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Startup probe error: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused... 57m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Container created 57m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Startup probe error: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Startup probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Startup probe error: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Startup probe failed: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused 57m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Startup probe failed: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused 57m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Startup probe error: Get "http://10.132.0.53:15021/healthz/ready": dial tcp 10.132.0.53:15021: connect: connection refused... 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 57m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled up replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 1 to 2 57m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Add eth0 [10.134.0.59/23] from ovn-kubernetes 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 57m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 57m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk 57m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Container created 57m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Container started 57m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused... 57m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused 56m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 1 to 2 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk 56m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Container started 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Add eth0 [10.132.0.62/23] from ovn-kubernetes 56m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Container created 56m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Startup probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Startup probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused 56m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled up replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 1 to 2 56m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 56m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 1 to 2 56m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 1 to 2 56m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Container started 56m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Container started 56m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Container created 56m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Add eth0 [10.134.0.62/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Container created 56m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Container started 56m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Container created 56m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Add eth0 [10.134.0.61/23] from ovn-kubernetes 56m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Add eth0 [10.132.0.64/23] from ovn-kubernetes 56m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Startup probe error: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Startup probe failed: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused 56m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 56m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 56m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml 56m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 56m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 1 to 2 56m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 2 to 3 56m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Add eth0 [10.133.0.54/23] from ovn-kubernetes 56m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Container started 56m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Add eth0 [10.134.0.65/23] from ovn-kubernetes 56m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Container created 56m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Container started 56m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 56m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Container created 56m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Startup probe error: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused... 56m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Startup probe failed: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused 56m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 55m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Container created 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr 55m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled up replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 1 to 3 55m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf 55m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Container started 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled up replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 2 to 4 55m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Add eth0 [10.132.0.65/23] from ovn-kubernetes 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Add eth0 [10.133.0.56/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Container created 55m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Container started 55m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Container created 55m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Add eth0 [10.133.0.57/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Add eth0 [10.132.0.66/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Container started 55m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Container started 55m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Container created 55m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Startup probe failed: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Startup probe error: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Startup probe error: Get "http://10.133.0.57:15021/healthz/ready": dial tcp 10.133.0.57:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Startup probe failed: Get "http://10.133.0.57:15021/healthz/ready": dial tcp 10.133.0.57:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Startup probe error: Get "http://10.132.0.66:15021/healthz/ready": dial tcp 10.132.0.66:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Startup probe failed: Get "http://10.132.0.66:15021/healthz/ready": dial tcp 10.132.0.66:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Startup probe failed: Get "http://10.133.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Startup probe error: Get "http://10.133.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Startup probe failed: Get "http://10.132.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Startup probe error: Get "http://10.132.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Startup probe error: Get "http://10.132.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Startup probe failed: Get "http://10.132.0.66:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 55m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 2 to 4 55m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Container created 55m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Container started 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled up replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 2 to 4 55m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Container started 55m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Container created 55m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Add eth0 [10.134.0.77/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Container started 55m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Add eth0 [10.132.0.67/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Add eth0 [10.133.0.60/23] from ovn-kubernetes 55m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Add eth0 [10.133.0.59/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 55m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Container created 55m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Container started 55m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Container created 55m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 2 to 3 55m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Add eth0 [10.134.0.78/23] from ovn-kubernetes 55m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn 55m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled up replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 4 to 5 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 55m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz 55m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 3 to 5 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 55m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 55m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Container started 55m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Container started 55m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Container created 55m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Container created 55m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Container started 55m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Add eth0 [10.132.0.69/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Container created 55m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Add eth0 [10.133.0.61/23] from ovn-kubernetes 55m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Add eth0 [10.134.0.79/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Container started 55m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Container created 55m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 55m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs 55m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 (combined from similar events): Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx 55m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt 55m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 55m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Add eth0 [10.134.0.81/23] from ovn-kubernetes 55m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Container created 55m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Container started 55m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Container created 55m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Container started 55m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Container created 55m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Add eth0 [10.134.0.80/23] from ovn-kubernetes 55m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Add eth0 [10.133.0.62/23] from ovn-kubernetes 55m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Container created 55m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Container started 55m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 55m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Container started 55m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Add eth0 [10.132.0.70/23] from ovn-kubernetes 55m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 55m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused... 55m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": dial tcp 10.134.0.79:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Startup probe error: Get "http://10.134.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 55m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Startup probe failed: Get "http://10.134.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 55m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 55m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 55m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 55m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 55m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Startup probe failed: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused 55m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Startup probe error: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused... 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 55m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 55m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 2 to 4 55m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Startup probe error: Get "http://10.134.0.81:15021/healthz/ready": dial tcp 10.134.0.81:15021: connect: connection refused... 55m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv 55m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Startup probe failed: Get "http://10.134.0.80:15021/healthz/ready": dial tcp 10.134.0.80:15021: connect: connection refused 55m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Container created 55m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Add eth0 [10.132.0.72/23] from ovn-kubernetes 55m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Container created 55m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Container started 55m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 55m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Add eth0 [10.133.0.64/23] from ovn-kubernetes 55m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Container started 55m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 55m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 55m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 55m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled up replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 3 to 6 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 54m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Container created 54m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Add eth0 [10.134.0.83/23] from ovn-kubernetes 54m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Add eth0 [10.132.0.74/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Container created 54m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Container started 54m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Container created 54m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Add eth0 [10.133.0.65/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Container created 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 54m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd 54m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Add eth0 [10.133.0.66/23] from ovn-kubernetes 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 54m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled up replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 4 to 7 54m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Container started 54m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Add eth0 [10.132.0.75/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Container started 54m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Add eth0 [10.134.0.84/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Container created 54m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Container started 54m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Container started 54m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Container created 54m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Container started 54m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Startup probe error: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Startup probe failed: Get "http://10.134.0.84:15021/healthz/ready": dial tcp 10.134.0.84:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Startup probe error: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Startup probe failed: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Startup probe failed: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Startup probe failed: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Startup probe error: Get "http://10.134.0.83:15021/healthz/ready": dial tcp 10.134.0.83:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Startup probe error: Get "http://10.132.0.74:15021/healthz/ready": dial tcp 10.132.0.74:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Startup probe failed: Get "http://10.134.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Startup probe error: Get "http://10.134.0.83:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 54m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 3 to 4 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 7; reason: 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj 54m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Container started 54m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g 54m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Container started 54m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Add eth0 [10.132.0.76/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled up replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 6 to 7 54m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Container created 54m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Container created 54m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Add eth0 [10.134.0.90/23] from ovn-kubernetes 54m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Startup probe error: Get "http://10.134.0.90:15021/healthz/ready": dial tcp 10.134.0.90:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Startup probe failed: Get "http://10.134.0.90:15021/healthz/ready": dial tcp 10.134.0.90:15021: connect: connection refused 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg 54m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 5 to 9 54m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 9; reason: cpu resource utilization (percentage of request) above target 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq 54m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft 54m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Add eth0 [10.133.0.74/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Container created 54m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Container started 54m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Add eth0 [10.132.0.77/23] from ovn-kubernetes 54m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Container started 54m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Container created 54m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Add eth0 [10.134.0.92/23] from ovn-kubernetes 54m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Container created 54m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Container started 54m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 54m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Add eth0 [10.133.0.73/23] from ovn-kubernetes 54m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Container created 54m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Container started 54m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Startup probe failed: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused 54m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Startup probe failed: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Startup probe error: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Startup probe error: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Startup probe failed: Get "http://10.134.0.92:15021/healthz/ready": dial tcp 10.134.0.92:15021: connect: connection refused 54m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Startup probe error: Get "http://10.132.0.77:15021/healthz/ready": dial tcp 10.132.0.77:15021: connect: connection refused... 54m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": dial tcp 10.133.0.73:15021: connect: connection refused... 54m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Startup probe failed: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 54m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Startup probe error: Get "http://10.133.0.73:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 4 to 5 53m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f 53m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Add eth0 [10.133.0.76/23] from ovn-kubernetes 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q 53m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Add eth0 [10.134.0.102/23] from ovn-kubernetes 53m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 4 to 7 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 53m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Container created 53m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Add eth0 [10.133.0.77/23] from ovn-kubernetes 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb 53m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Add eth0 [10.132.0.80/23] from ovn-kubernetes 53m Normal SuccessfulCreate replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf (combined from similar events): Created pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp 53m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c 53m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 53m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Container started 53m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Container created 53m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Container created 53m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Add eth0 [10.134.0.101/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 53m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Container started 53m Normal AddedInterface pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Add eth0 [10.132.0.79/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Container created 53m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Container started 53m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled up replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 7 to 10 53m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Container created 53m Normal Started pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Container started 53m Normal Created pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Container created 53m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Add eth0 [10.134.0.100/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Container started 53m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Container started 53m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Container created 53m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Container started 53m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Startup probe failed: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Startup probe error: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Startup probe failed: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Startup probe failed: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Startup probe error: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Startup probe failed: Get "http://10.134.0.100:15021/healthz/ready": dial tcp 10.134.0.100:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Startup probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Startup probe error: Get "http://10.132.0.80:15021/healthz/ready": dial tcp 10.132.0.80:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Startup probe error: Get "http://10.134.0.101:15021/healthz/ready": dial tcp 10.134.0.101:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Startup probe error: Get "http://10.134.0.102:15021/healthz/ready": dial tcp 10.134.0.102:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Startup probe failed: Get "http://10.134.0.102:15021/healthz/ready": dial tcp 10.134.0.102:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Startup probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 53m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 4 to 8 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d 53m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Container created 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 (combined from similar events): Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 53m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Add eth0 [10.133.0.78/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Add eth0 [10.132.0.82/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Container created 53m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Container started 53m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Add eth0 [10.132.0.81/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Container created 53m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Container started 53m Normal SuccessfulCreate replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Created pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 53m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 5 to 7 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 53m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Container started 53m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Container created 53m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Add eth0 [10.133.0.79/23] from ovn-kubernetes 53m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled up replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 7 to 10 53m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Container started 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q 53m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m 53m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Container started 53m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Container started 53m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Container started 53m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Add eth0 [10.132.0.84/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Add eth0 [10.134.0.105/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Add eth0 [10.133.0.80/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Container created 53m Normal Started pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Container started 53m Normal AddedInterface pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Add eth0 [10.132.0.83/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Container created 53m Normal Pulled pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Container created 53m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Container created 53m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Container started 53m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Container created 53m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Add eth0 [10.134.0.106/23] from ovn-kubernetes 53m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Startup probe error: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Startup probe error: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Startup probe error: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Startup probe failed: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Startup probe failed: Get "http://10.133.0.78:15021/healthz/ready": dial tcp 10.133.0.78:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Startup probe failed: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Startup probe error: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Startup probe failed: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Startup probe failed: Get "http://10.134.0.105:15021/healthz/ready": dial tcp 10.134.0.105:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Startup probe error: Get "http://10.134.0.106:15021/healthz/ready": dial tcp 10.134.0.106:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 (combined from similar events): Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l 53m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled up replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 9 to 10 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm 53m Normal SuccessfulCreate replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 (combined from similar events): Created pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v 53m Normal SuccessfulCreate replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Created pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 53m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled up replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 5 to 10 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 53m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 8 to 9 53m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 53m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 9; reason: 53m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Add eth0 [10.133.0.82/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Add eth0 [10.134.0.109/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Add eth0 [10.133.0.81/23] from ovn-kubernetes 53m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Add eth0 [10.133.0.83/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Container started 53m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Add eth0 [10.134.0.110/23] from ovn-kubernetes 53m Normal AddedInterface pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Add eth0 [10.132.0.85/23] from ovn-kubernetes 53m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Container created 53m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Container created 53m Normal AddedInterface pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Add eth0 [10.132.0.86/23] from ovn-kubernetes 53m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Container started 53m Normal Pulled pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Pulled pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 53m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Container created 53m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Container started 53m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Container started 53m Normal Created pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Container created 53m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Container created 53m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Container started 53m Normal Started pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Container started 53m Normal Started pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Container started 53m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Container created 53m Normal Created pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Container created 53m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Startup probe error: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Startup probe error: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Startup probe error: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Startup probe failed: Get "http://10.133.0.83:15021/healthz/ready": dial tcp 10.133.0.83:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Startup probe failed: Get "http://10.134.0.110:15021/healthz/ready": dial tcp 10.134.0.110:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Startup probe failed: Get "http://10.132.0.85:15021/healthz/ready": dial tcp 10.132.0.85:15021: connect: connection refused 53m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Startup probe error: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused... 53m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Startup probe error: Get "http://10.134.0.109:15021/healthz/ready": dial tcp 10.134.0.109:15021: connect: connection refused... 53m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Startup probe failed: Get "http://10.132.0.86:15021/healthz/ready": dial tcp 10.132.0.86:15021: connect: connection refused 53m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Startup probe failed: Get "http://10.134.0.109:15021/healthz/ready": dial tcp 10.134.0.109:15021: connect: connection refused 53m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 53m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 53m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-nxlpl 53m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 53m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Add eth0 [10.134.0.115/23] from ovn-kubernetes 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 (combined from similar events): Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk 52m Normal SuccessfulCreate replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Created pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd 52m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled up replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 7 to 10 52m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Readiness probe failed: Get "http://10.134.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 52m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Readiness probe error: Get "http://10.134.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 52m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Add eth0 [10.133.0.84/23] from ovn-kubernetes 52m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Container created 52m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Container started 52m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Add eth0 [10.132.0.88/23] from ovn-kubernetes 52m Normal AddedInterface pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Add eth0 [10.134.0.116/23] from ovn-kubernetes 52m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Container created 52m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Container started 52m Normal Started pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Container started 52m Normal Created pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Container created 52m Normal Pulled pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Container created 52m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 12.218s (12.218s including waiting). Image size: 185476477 bytes. 52m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Container started 52m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Readiness probe error: HTTP probe failed with statuscode: 503... 52m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Startup probe failed: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Startup probe error: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused... 52m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Readiness probe failed: HTTP probe failed with statuscode: 503 52m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Startup probe failed: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Startup probe error: Get "http://10.132.0.88:15021/healthz/ready": dial tcp 10.132.0.88:15021: connect: connection refused... 52m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Startup probe error: Get "http://10.134.0.116:15021/healthz/ready": dial tcp 10.134.0.116:15021: connect: connection refused... 52m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Startup probe failed: Get "http://10.134.0.116:15021/healthz/ready": dial tcp 10.134.0.116:15021: connect: connection refused 52m Normal SuccessfulCreate replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf (combined from similar events): Created pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t 52m Normal Created pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Container created 52m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 52m Normal Pulled pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 52m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled up replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 9 to 10 52m Normal AddedInterface pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Add eth0 [10.132.0.89/23] from ovn-kubernetes 52m Normal Started pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Container started 52m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Startup probe failed: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused 52m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Startup probe error: Get "http://10.132.0.89:15021/healthz/ready": dial tcp 10.132.0.89:15021: connect: connection refused... 51m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Readiness probe failed: Get "http://10.134.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Readiness probe error: Get "http://10.134.0.43:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Normal AddedLabel gateway/e2e-ait-d67de78b Added label istio.io/rev=openshift-gateway to gateway e2e-ait-d67de78b 51m Normal SuccessfulCreate replicaset/e2e-ait-d67de78b-openshift-default-7955fc99c9 Created pod: e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx 51m Normal ScalingReplicaSet deployment/e2e-ait-d67de78b-openshift-default Scaled up replica set e2e-ait-d67de78b-openshift-default-7955fc99c9 from 0 to 1 51m Normal Pulled pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 51m Normal Created pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Container created 51m Normal AddedInterface pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Add eth0 [10.132.0.90/23] from ovn-kubernetes 51m Normal Started pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Container started 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-d67de78b-c65b577cb Created pod: payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-d67de78b Scaled up replica set payload-pre-processing-e2e-ait-d67de78b-767d544f6c from 0 to 1 51m Warning FailedCreate replicaset/payload-processing-e2e-ait-d67de78b-c65b577cb Error creating: pods "payload-processing-e2e-ait-d67de78b-c65b577cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-d67de78b: serviceaccount "payload-processing-e2e-ait-d67de78b" not found 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-d67de78b Scaled up replica set payload-processing-e2e-ait-d67de78b-c65b577cb from 0 to 1 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-d67de78b-767d544f6c Created pod: payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-d67de78b-767d544f6c Error creating: pods "payload-pre-processing-e2e-ait-d67de78b-767d544f6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-d67de78b: serviceaccount "payload-processing-e2e-ait-d67de78b" not found 51m Normal Started pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Container started 51m Normal AddedInterface pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Add eth0 [10.133.0.94/23] from ovn-kubernetes 51m Normal Pulled pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Container created 51m Normal Created pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Container created 51m Normal Started pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Container started 51m Normal AddedInterface pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Add eth0 [10.132.0.91/23] from ovn-kubernetes 51m Normal Pulled pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal SuccessfulCreate replicaset/e2e-ait-b0fba292-openshift-default-658566db69 Created pod: e2e-ait-b0fba292-openshift-default-658566db69-7r6dr 51m Normal ScalingReplicaSet deployment/e2e-ait-b0fba292-openshift-default Scaled up replica set e2e-ait-b0fba292-openshift-default-658566db69 from 0 to 1 51m Normal AddedLabel gateway/e2e-ait-b0fba292 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-b0fba292 51m Normal AddedInterface pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Add eth0 [10.132.0.92/23] from ovn-kubernetes 51m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-b0fba292 Scaled up replica set payload-processing-e2e-ait-b0fba292-65679f777d from 0 to 1 51m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-b0fba292 Scaled up replica set payload-pre-processing-e2e-ait-b0fba292-7bf96c589f from 0 to 1 51m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f Created pod: payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp 51m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f Error creating: pods "payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b0fba292: serviceaccount "payload-processing-e2e-ait-b0fba292" not found 51m Warning FailedCreate replicaset/payload-processing-e2e-ait-b0fba292-65679f777d Error creating: pods "payload-processing-e2e-ait-b0fba292-65679f777d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b0fba292: serviceaccount "payload-processing-e2e-ait-b0fba292" not found 51m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-b0fba292-65679f777d Created pod: payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 51m Normal Started pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Container started 51m Normal Started pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Container started 51m Normal Pulled pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal AddedInterface pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Add eth0 [10.133.0.95/23] from ovn-kubernetes 51m Normal AddedInterface pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Add eth0 [10.132.0.93/23] from ovn-kubernetes 51m Normal Pulled pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 51m Normal Created pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Container created 51m Normal Created pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Container created 51m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Readiness probe error: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Readiness probe failed: Get "http://10.134.0.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Readiness probe error: Get "http://10.132.0.53:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Readiness probe failed: Get "http://10.132.0.53:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Readiness probe failed: Get "http://10.134.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Readiness probe error: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Readiness probe failed: Get "http://10.132.0.88:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Readiness probe failed: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Readiness probe error: Get "http://10.132.0.86:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Readiness probe error: Get "http://10.134.0.116:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-b0fba292-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-b0fba292-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-d67de78b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-d67de78b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 51m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ait-b0fba292-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ait-b0fba292-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 51m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Readiness probe error: Get "http://10.132.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Readiness probe failed: Get "http://10.132.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 51m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Readiness probe error: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 51m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Readiness probe failed: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Normal AddedLabel gateway/e2e-shared-a-w3-097d90 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w3-097d90 50m Normal SuccessfulCreate replicaset/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7 Created pod: e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms 50m Normal ScalingReplicaSet deployment/e2e-shared-a-w3-097d90-openshift-default Scaled up replica set e2e-shared-a-w3-097d90-openshift-default-64f69f59b7 from 0 to 1 50m Normal AddedInterface pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Add eth0 [10.132.0.95/23] from ovn-kubernetes 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w3-097d90 Scaled up replica set payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67 from 0 to 1 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w3-097d90 Scaled up replica set payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb Created pod: payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 50m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb Error creating: pods "payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-097d90: serviceaccount "payload-processing-e2e-shared-a-w3-097d90" not found 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67 Error creating: pods "payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w3-097d90: serviceaccount "payload-processing-e2e-shared-a-w3-097d90" not found 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67 Created pod: payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 50m Normal Started pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Container started 50m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Container created 50m Normal Pulled pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Add eth0 [10.132.0.96/23] from ovn-kubernetes 50m Normal Started pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Container started 50m Normal AddedInterface pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Add eth0 [10.132.0.97/23] from ovn-kubernetes 50m Normal Created pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Container created 50m Warning Unhealthy pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Readiness probe failed: timeout: health rpc did not complete within 1s 50m Normal Killing pod/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b Stopping container payload-processing 50m Normal Killing pod/payload-pre-processing-e2e-ait-d67de78b-767d544f6c-2v6s6 Stopping container payload-pre-processing 50m Normal Killing pod/payload-pre-processing-e2e-ait-b0fba292-7bf96c589f-x9vcp Stopping container payload-pre-processing 50m Normal Killing pod/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74 Stopping container payload-processing 50m Normal Killing pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Stopping container istio-proxy 50m Normal AddedLabel gateway/e2e-migrate-5a926a9d-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-5a926a9d-gw 50m Normal ScalingReplicaSet deployment/e2e-migrate-5a926a9d-gw-openshift-default Scaled up replica set e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777 from 0 to 1 50m Normal SuccessfulCreate replicaset/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777 Created pod: e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk 50m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-ft7s9 50m Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 50m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Add eth0 [10.133.0.102/23] from ovn-kubernetes 50m Normal AddedInterface pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Add eth0 [10.133.0.101/23] from ovn-kubernetes 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-5a926a9d Scaled up replica set payload-pre-processing-e2e-migrate-5a926a9d-df64f747 from 0 to 1 50m Warning ProbeError pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Readiness probe error: Get "http://10.132.0.90:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning Unhealthy pod/e2e-ait-d67de78b-openshift-default-7955fc99c9-szsbx Readiness probe failed: Get "http://10.132.0.90:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning FailedCreate replicaset/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4 Error creating: pods "payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-5a926a9d: serviceaccount "payload-processing-e2e-migrate-5a926a9d" not found 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-5a926a9d-df64f747 Error creating: pods "payload-pre-processing-e2e-migrate-5a926a9d-df64f747-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-5a926a9d: serviceaccount "payload-processing-e2e-migrate-5a926a9d" not found 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-5a926a9d Scaled up replica set payload-processing-e2e-migrate-5a926a9d-dcb9b57f4 from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4 Created pod: payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj 50m Normal Pulled pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-5a926a9d-df64f747 Created pod: payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk 50m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Add eth0 [10.134.0.134/23] from ovn-kubernetes 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 7 to 8 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 50m Normal AddedInterface pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Add eth0 [10.132.0.98/23] from ovn-kubernetes 50m Normal Started pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Container started 50m Normal Pulled pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Started pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Container started 50m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Container created 50m Normal Created pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Container created 50m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Add eth0 [10.134.0.135/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Container started 50m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Container created 50m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Container created 50m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Container started 50m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 5.127s (5.127s including waiting). Image size: 185476477 bytes. 50m Normal SuccessfulCreate replicaset/e2e-ait-b0fba292-openshift-default-658566db69 Created pod: e2e-ait-b0fba292-openshift-default-658566db69-qx79q 50m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Readiness probe error: HTTP probe failed with statuscode: 503... 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ait-b0fba292-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 50m Normal ScalingReplicaSet deployment/e2e-ait-b0fba292-openshift-default Scaled up replica set e2e-ait-b0fba292-openshift-default-658566db69 from 1 to 3 50m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 Readiness probe failed: HTTP probe failed with statuscode: 503 50m Normal SuccessfulCreate replicaset/e2e-ait-b0fba292-openshift-default-658566db69 Created pod: e2e-ait-b0fba292-openshift-default-658566db69-l8bcz 50m Normal Pulled pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Created pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Container created 50m Normal AddedInterface pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Add eth0 [10.133.0.103/23] from ovn-kubernetes 50m Normal Pulled pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal Pulled pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Add eth0 [10.134.0.136/23] from ovn-kubernetes 50m Normal Created pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Container created 50m Normal Started pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Container started 50m Normal Started pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Container started 50m Normal Started pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Container started 50m Normal Created pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Container created 50m Warning ProbeError pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Startup probe error: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Startup probe failed: Get "http://10.134.0.136:15021/healthz/ready": dial tcp 10.134.0.136:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Startup probe failed: Get "http://10.133.0.103:15021/healthz/ready": dial tcp 10.133.0.103:15021: connect: connection refused 50m Warning ProbeError pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Startup probe error: Get "http://10.134.0.136:15021/healthz/ready": dial tcp 10.134.0.136:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Startup probe error: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused... 50m Normal Killing pod/e2e-ait-b0fba292-openshift-default-658566db69-qx79q Stopping container istio-proxy 50m Normal Killing pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz Stopping container istio-proxy 50m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Startup probe error: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused... 50m Normal Killing pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Stopping container istio-proxy 50m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Startup probe failed: Get "http://10.134.0.135:15021/healthz/ready": dial tcp 10.134.0.135:15021: connect: connection refused 50m Warning Unhealthy pod/e2e-ait-b0fba292-openshift-default-658566db69-7r6dr Startup probe failed: Get "http://10.132.0.92:15021/healthz/ready": dial tcp 10.132.0.92:15021: connect: connection refused 50m Normal AddedLabel gateway/e2e-derive-47b1fc31 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-47b1fc31 50m Normal SuccessfulCreate replicaset/e2e-derive-47b1fc31-openshift-default-59b9d89477 Created pod: e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb 50m Normal ScalingReplicaSet deployment/e2e-derive-47b1fc31-openshift-default Scaled up replica set e2e-derive-47b1fc31-openshift-default-59b9d89477 from 0 to 1 50m Normal AddedInterface pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Add eth0 [10.134.0.137/23] from ovn-kubernetes 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-5a926a9d-gw-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-5a926a9d-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 50m Warning FailedCreate replicaset/payload-processing-e2e-derive-47b1fc31-5c7757d9cd Error creating: pods "payload-processing-e2e-derive-47b1fc31-5c7757d9cd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-47b1fc31: serviceaccount "payload-processing-e2e-derive-47b1fc31" not found 50m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-47b1fc31 Scaled up replica set payload-pre-processing-e2e-derive-47b1fc31-d4d495d95 from 0 to 1 50m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95 Created pod: payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 50m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-47b1fc31-5c7757d9cd Created pod: payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d 50m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-47b1fc31 Scaled up replica set payload-processing-e2e-derive-47b1fc31-5c7757d9cd from 0 to 1 50m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95 Error creating: pods "payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-47b1fc31: serviceaccount "payload-processing-e2e-derive-47b1fc31" not found 50m Normal AddedInterface pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Add eth0 [10.132.0.100/23] from ovn-kubernetes 50m Normal Started pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Container started 50m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd 50m Normal AddedInterface pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Add eth0 [10.134.0.138/23] from ovn-kubernetes 50m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled up replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 8 to 10 50m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Add eth0 [10.133.0.104/23] from ovn-kubernetes 50m Normal Pulled pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal Created pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Container created 50m Normal Started pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Container started 50m Normal Created pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Container created 50m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 50m Normal Pulled pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 50m Normal SuccessfulCreate replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 (combined from similar events): Created pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 50m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Container started 50m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Container created 50m Normal Pulled pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 50m Normal AddedInterface pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Add eth0 [10.132.0.101/23] from ovn-kubernetes 50m Normal Started pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Container started 50m Normal Created pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Container created 50m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Startup probe failed: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Startup probe error: Get "http://10.133.0.104:15021/healthz/ready": dial tcp 10.133.0.104:15021: connect: connection refused... 50m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Startup probe failed: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused 50m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Startup probe error: Get "http://10.132.0.101:15021/healthz/ready": dial tcp 10.132.0.101:15021: connect: connection refused... 50m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Startup probe error: Get "http://10.132.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 50m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-migrate-5a926a9d-gw-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 50m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Startup probe failed: Get "http://10.132.0.101:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 50m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-migrate-5a926a9d-gw-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7 Created pod: e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 49m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-e15c99-openshift-default Scaled up replica set e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7 from 0 to 1 49m Normal AddedLabel gateway/e2e-shared-b-w3-e15c99 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w3-e15c99 49m Normal AddedInterface pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Add eth0 [10.134.0.139/23] from ovn-kubernetes 49m Normal Pulled pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Container created 49m Normal Started pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Container started 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w3-e15c99 Scaled up replica set payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w3-e15c99 Scaled up replica set payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f Created pod: payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb Error creating: pods "payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99: serviceaccount "payload-processing-e2e-shared-b-w3-e15c99" not found 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f Error creating: pods "payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99: serviceaccount "payload-processing-e2e-shared-b-w3-e15c99" not found 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb Created pod: payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf 49m Normal AddedInterface pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Add eth0 [10.133.0.105/23] from ovn-kubernetes 49m Normal Created pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Container created 49m Normal Started pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Container started 49m Normal Pulled pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Container started 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Add eth0 [10.132.0.104/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Container created 49m Warning ProbeError pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Startup probe error: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Startup probe failed: Get "http://10.132.0.95:15021/healthz/ready": dial tcp 10.132.0.95:15021: connect: connection refused 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-097d90-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-097d90-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Normal AddedLabel gateway/e2e-mt-c613f5ea Added label istio.io/rev=openshift-gateway to gateway e2e-mt-c613f5ea 49m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s 49m Normal ScalingReplicaSet deployment/e2e-mt-c613f5ea-openshift-default Scaled up replica set e2e-mt-c613f5ea-openshift-default-55cfff77f7 from 0 to 1 49m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Add eth0 [10.134.0.142/23] from ovn-kubernetes 49m Normal Killing pod/payload-pre-processing-e2e-migrate-5a926a9d-df64f747-cpjjk Stopping container payload-pre-processing 49m Normal Killing pod/payload-processing-e2e-migrate-5a926a9d-dcb9b57f4-2hm9q Stopping container payload-processing 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-c613f5ea Scaled up replica set payload-processing-e2e-mt-c613f5ea-6c844f4556 from 0 to 1 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-c613f5ea Scaled up replica set payload-pre-processing-e2e-mt-c613f5ea-55445d94bc from 0 to 1 49m Normal Created pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Container created 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-c613f5ea-6c844f4556 Created pod: payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 49m Normal Pulled pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Container created 49m Normal Pulled pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal Pulled pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 49m Normal AddedInterface pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Add eth0 [10.133.0.106/23] from ovn-kubernetes 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc Error creating: pods "payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c613f5ea: serviceaccount "payload-processing-e2e-mt-c613f5ea" not found 49m Normal Started pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Container started 49m Warning FailedCreate replicaset/payload-processing-e2e-mt-c613f5ea-6c844f4556 Error creating: pods "payload-processing-e2e-mt-c613f5ea-6c844f4556-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c613f5ea: serviceaccount "payload-processing-e2e-mt-c613f5ea" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc Created pod: payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj 49m Normal Created pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Container created 49m Normal Created pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Container created 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w3-097d90-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w3-097d90-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Normal Started pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Container started 49m Normal Started pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Container started 49m Normal Pulled pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Started pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Container started 49m Normal AddedInterface pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Add eth0 [10.134.0.143/23] from ovn-kubernetes 49m Warning ProbeError pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Startup probe error: Get "http://10.134.0.137:15021/healthz/ready": dial tcp 10.134.0.137:15021: connect: connection refused... 49m Warning Unhealthy pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Startup probe failed: Get "http://10.134.0.137:15021/healthz/ready": dial tcp 10.134.0.137:15021: connect: connection refused 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-47b1fc31-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-47b1fc31-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 49m Warning ProbeError pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Startup probe error: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused... 49m Normal Killing pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Stopping container istio-proxy 49m Warning Unhealthy pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk Startup probe failed: Get "http://10.133.0.101:15021/healthz/ready": dial tcp 10.133.0.101:15021: connect: connection refused 49m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9 Created pod: e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c 49m Normal AddedLabel gateway/e2e-shared-a-w2-44bc69 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-44bc69 49m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-44bc69-openshift-default Scaled up replica set e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9 from 0 to 1 49m Normal AddedInterface pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Add eth0 [10.132.0.106/23] from ovn-kubernetes 49m Normal Killing pod/payload-processing-e2e-derive-47b1fc31-5c7757d9cd-dvf6d Stopping container payload-processing 49m Normal Killing pod/payload-pre-processing-e2e-derive-47b1fc31-d4d495d95-knhk7 Stopping container payload-pre-processing 49m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-44bc69 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc from 0 to 1 49m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b Created pod: payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q 49m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc Error creating: pods "payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69: serviceaccount "payload-processing-e2e-shared-a-w2-44bc69" not found 49m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b Error creating: pods "payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69: serviceaccount "payload-processing-e2e-shared-a-w2-44bc69" not found 49m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc Created pod: payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st 49m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-44bc69 Scaled up replica set payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b from 0 to 1 49m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Add eth0 [10.133.0.108/23] from ovn-kubernetes 49m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Container started 49m Normal Started pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Container started 49m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Add eth0 [10.133.0.107/23] from ovn-kubernetes 49m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Container created 49m Normal Pulled pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 49m Normal Created pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Container created 49m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-derive-47b1fc31-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-derive-47b1fc31-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 49m Normal Created pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Container created 49m Normal Started pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Container started 49m Normal Pulled pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Normal Killing pod/e2e-derive-47b1fc31-openshift-default-59b9d89477-v4rrb Stopping container istio-proxy 48m Normal AddedLabel gateway/e2e-shared-a-w0-ff4760 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w0-ff4760 48m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757 Created pod: e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j 48m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ff4760-openshift-default Scaled up replica set e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757 from 0 to 1 48m Normal AddedInterface pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Add eth0 [10.134.0.145/23] from ovn-kubernetes 48m Warning ProbeError pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Startup probe error: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Startup probe failed: Get "http://10.134.0.139:15021/healthz/ready": dial tcp 10.134.0.139:15021: connect: connection refused 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w0-ff4760 Scaled up replica set payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9 from 0 to 1 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9 Error creating: pods "payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760: serviceaccount "payload-processing-e2e-shared-a-w0-ff4760" not found 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb Created pod: payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9 Created pod: payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl 48m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb Error creating: pods "payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w0-ff4760: serviceaccount "payload-processing-e2e-shared-a-w0-ff4760" not found 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w0-ff4760 Scaled up replica set payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb from 0 to 1 48m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Add eth0 [10.133.0.109/23] from ovn-kubernetes 48m Normal AddedInterface pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Add eth0 [10.132.0.108/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Container created 48m Normal Started pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Container started 48m Normal Started pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Container started 48m Normal Created pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Container created 48m Normal Pulled pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-44bc69-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-44bc69-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-e15c99-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-e15c99-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w3-e15c99-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w3-e15c99-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Readiness probe failed: Get "http://10.134.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 48m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Readiness probe error: Get "http://10.134.0.142:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 48m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Container started 48m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Container created 48m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 48m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Startup probe error: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused... 48m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Startup probe failed: Get "http://10.134.0.142:15021/healthz/ready": dial tcp 10.134.0.142:15021: connect: connection refused 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 48m Normal AddedLabel gateway/e2e-shared-b-w2-93bad9 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-93bad9 48m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled up replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 0 to 1 48m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n 48m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Add eth0 [10.134.0.146/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-93bad9 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-93bad9 Scaled up replica set payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8 from 0 to 1 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8 Created pod: payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf Error creating: pods "payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9: serviceaccount "payload-processing-e2e-shared-b-w2-93bad9" not found 48m Normal AddedLabel gateway/e2e-api-a-528d67 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-528d67 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf Created pod: payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk 48m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8 Error creating: pods "payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-93bad9: serviceaccount "payload-processing-e2e-shared-b-w2-93bad9" not found 48m Normal Created pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Container created 48m Normal Started pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Container started 48m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Container created 48m Normal ScalingReplicaSet deployment/e2e-api-a-528d67-openshift-default Scaled up replica set e2e-api-a-528d67-openshift-default-5c45549d45 from 0 to 1 48m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Add eth0 [10.132.0.110/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Pulled pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Container started 48m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Add eth0 [10.134.0.147/23] from ovn-kubernetes 48m Normal SuccessfulCreate replicaset/e2e-api-a-528d67-openshift-default-5c45549d45 Created pod: e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm 48m Normal AddedInterface pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Add eth0 [10.134.0.148/23] from ovn-kubernetes 48m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-528d67 Scaled up replica set payload-pre-processing-e2e-api-a-528d67-654d5c4795 from 0 to 1 48m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-528d67 Scaled up replica set payload-processing-e2e-api-a-528d67-5998c8d466 from 0 to 1 48m Warning FailedCreate replicaset/payload-processing-e2e-api-a-528d67-5998c8d466 Error creating: pods "payload-processing-e2e-api-a-528d67-5998c8d466-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-528d67: serviceaccount "payload-processing-e2e-api-a-528d67" not found 48m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-528d67-654d5c4795 Created pod: payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj 48m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-528d67-5998c8d466 Created pod: payload-processing-e2e-api-a-528d67-5998c8d466-84qzk 48m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-528d67-654d5c4795 Error creating: pods "payload-pre-processing-e2e-api-a-528d67-654d5c4795-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-528d67: serviceaccount "payload-processing-e2e-api-a-528d67" not found 48m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal Created pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Container created 48m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Add eth0 [10.134.0.149/23] from ovn-kubernetes 48m Normal Pulled pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Normal Created pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Container created 48m Normal Started pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Container started 48m Normal AddedInterface pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Add eth0 [10.134.0.150/23] from ovn-kubernetes 48m Normal Pulled pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 48m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 48m Normal Started pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Container started 47m Normal AddedLabel gateway/e2e-mt-ef844616 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-ef844616 47m Normal SuccessfulCreate replicaset/e2e-mt-ef844616-openshift-default-f57fcfc4f Created pod: e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc 47m Normal ScalingReplicaSet deployment/e2e-mt-ef844616-openshift-default Scaled up replica set e2e-mt-ef844616-openshift-default-f57fcfc4f from 0 to 1 47m Normal AddedInterface pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Add eth0 [10.133.0.112/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-dfb231-openshift-default Scaled up replica set e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 from 0 to 1 47m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 Created pod: e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg 47m Normal AddedLabel gateway/e2e-shared-b-w0-dfb231 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w0-dfb231 47m Normal AddedInterface pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Add eth0 [10.132.0.111/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-ef844616 Scaled up replica set payload-processing-e2e-mt-ef844616-5cb9d44b8 from 0 to 1 47m Warning FailedCreate replicaset/payload-processing-e2e-mt-ef844616-5cb9d44b8 Error creating: pods "payload-processing-e2e-mt-ef844616-5cb9d44b8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ef844616: serviceaccount "payload-processing-e2e-mt-ef844616" not found 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-ef844616 Scaled up replica set payload-pre-processing-e2e-mt-ef844616-7b44b7cd87 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-ef844616-5cb9d44b8 Created pod: payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87 Created pod: payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87 Error creating: pods "payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-ef844616: serviceaccount "payload-processing-e2e-mt-ef844616" not found 47m Normal Created pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Container created 47m Normal AddedInterface pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Add eth0 [10.132.0.112/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedLabel gateway/e2e-inf-a-ac96f3 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-ac96f3 47m Normal Started pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Container started 47m Normal Started pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Container started 47m Normal Created pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Container created 47m Normal Pulled pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Add eth0 [10.134.0.151/23] from ovn-kubernetes 47m Normal AddedInterface pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Add eth0 [10.133.0.113/23] from ovn-kubernetes 47m Normal SuccessfulCreate replicaset/e2e-inf-a-ac96f3-openshift-default-86c6f75967 Created pod: e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 47m Normal ScalingReplicaSet deployment/e2e-inf-a-ac96f3-openshift-default Scaled up replica set e2e-inf-a-ac96f3-openshift-default-86c6f75967 from 0 to 1 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w0-dfb231 Scaled up replica set payload-processing-e2e-shared-b-w0-dfb231-b5867f45d from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d Created pod: payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w0-dfb231 Scaled up replica set payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7 Created pod: payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl 47m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d Error creating: pods "payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231: serviceaccount "payload-processing-e2e-shared-b-w0-dfb231" not found 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7 Error creating: pods "payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w0-dfb231: serviceaccount "payload-processing-e2e-shared-b-w0-dfb231" not found 47m Normal Pulled pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Add eth0 [10.132.0.114/23] from ovn-kubernetes 47m Normal AddedInterface pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Add eth0 [10.132.0.113/23] from ovn-kubernetes 47m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Container started 47m Normal Created pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Container created 47m Normal Created pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Container created 47m Normal Started pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Container started 47m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 47m Normal ScalingReplicaSet deployment/e2e-mt-c613f5ea-openshift-default Scaled up replica set e2e-mt-c613f5ea-openshift-default-55cfff77f7 from 1 to 2 47m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Container created 47m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Container started 47m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799 Error creating: pods "payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-ac96f3: serviceaccount "payload-processing-e2e-inf-a-ac96f3" not found 47m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-ac96f3-55cf8f4776 Error creating: pods "payload-processing-e2e-inf-a-ac96f3-55cf8f4776-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-ac96f3: serviceaccount "payload-processing-e2e-inf-a-ac96f3" not found 47m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-ac96f3 Scaled up replica set payload-processing-e2e-inf-a-ac96f3-55cf8f4776 from 0 to 1 47m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Add eth0 [10.133.0.114/23] from ovn-kubernetes 47m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-ac96f3 Scaled up replica set payload-pre-processing-e2e-inf-a-ac96f3-f75f69799 from 0 to 1 47m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799 Created pod: payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch 47m Normal Created pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Container created 47m Normal Pulled pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 47m Normal Started pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Container started 47m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-ac96f3-55cf8f4776 Created pod: payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl 47m Normal AddedInterface pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Add eth0 [10.134.0.152/23] from ovn-kubernetes 47m Normal Created pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Container created 47m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Add eth0 [10.134.0.153/23] from ovn-kubernetes 47m Normal Started pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Container started 47m Normal Pulled pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Pulled pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 47m Normal Started pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Container started 47m Normal Created pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Container created 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-ef844616-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-ef844616-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Startup probe failed: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused 47m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Startup probe error: Get "http://10.133.0.114:15021/healthz/ready": dial tcp 10.133.0.114:15021: connect: connection refused... 47m Warning Unhealthy pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Startup probe failed: Get "http://10.134.0.145:15021/healthz/ready": dial tcp 10.134.0.145:15021: connect: connection refused 47m Warning ProbeError pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Startup probe error: Get "http://10.134.0.145:15021/healthz/ready": dial tcp 10.134.0.145:15021: connect: connection refused... 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-ac96f3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-ac96f3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-ff4760-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-ff4760-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 47m Normal Killing pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Stopping container payload-processing 47m Normal Killing pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Stopping container payload-pre-processing 47m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w0-ff4760-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 47m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w0-ff4760-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal AddedLabel gateway/e2e-api-a-543d08 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-543d08 46m Normal ScalingReplicaSet deployment/e2e-api-a-543d08-openshift-default Scaled up replica set e2e-api-a-543d08-openshift-default-f45f9c965 from 0 to 1 46m Normal AddedLabel gateway/e2e-api-b-645aef Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-645aef 46m Normal SuccessfulCreate replicaset/e2e-api-a-543d08-openshift-default-f45f9c965 Created pod: e2e-api-a-543d08-openshift-default-f45f9c965-597sx 46m Normal ScalingReplicaSet deployment/e2e-api-b-645aef-openshift-default Scaled up replica set e2e-api-b-645aef-openshift-default-7fd469db85 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-api-b-645aef-openshift-default-7fd469db85 Created pod: e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g 46m Normal AddedInterface pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Add eth0 [10.134.0.156/23] from ovn-kubernetes 46m Normal AddedInterface pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Add eth0 [10.134.0.157/23] from ovn-kubernetes 46m Warning Unhealthy pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 46m Warning Unhealthy pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-rvdv2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-543d08-7844bd78d7 Error creating: pods "payload-pre-processing-e2e-api-a-543d08-7844bd78d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-543d08: serviceaccount "payload-processing-e2e-api-a-543d08" not found 46m Warning ProbeError pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Readiness probe error: Get "http://10.134.0.148:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 46m Warning Unhealthy pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Readiness probe failed: Get "http://10.134.0.148:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-543d08 Scaled up replica set payload-pre-processing-e2e-api-a-543d08-7844bd78d7 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-543d08-7844bd78d7 Created pod: payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-543d08-6787968b78 Created pod: payload-processing-e2e-api-a-543d08-6787968b78-jpdvp 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-543d08 Scaled up replica set payload-processing-e2e-api-a-543d08-6787968b78 from 0 to 1 46m Warning FailedCreate replicaset/payload-processing-e2e-api-a-543d08-6787968b78 Error creating: pods "payload-processing-e2e-api-a-543d08-6787968b78-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-543d08: serviceaccount "payload-processing-e2e-api-a-543d08" not found 46m Normal Pulled pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Started pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Container started 46m Normal Started pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Container started 46m Normal Created pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Container created 46m Normal Created pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Container created 46m Normal Pulled pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Add eth0 [10.132.0.117/23] from ovn-kubernetes 46m Normal Started pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Container started 46m Normal Created pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Container created 46m Normal Pulled pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Add eth0 [10.132.0.116/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-645aef-59b6d9d59 Created pod: payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-645aef Scaled up replica set payload-processing-e2e-api-b-645aef-5b789579fd from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-645aef Scaled up replica set payload-pre-processing-e2e-api-b-645aef-59b6d9d59 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-645aef-5b789579fd Created pod: payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs 46m Warning FailedCreate replicaset/payload-processing-e2e-api-b-645aef-5b789579fd Error creating: pods "payload-processing-e2e-api-b-645aef-5b789579fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-645aef: serviceaccount "payload-processing-e2e-api-b-645aef" not found 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-645aef-59b6d9d59 Error creating: pods "payload-pre-processing-e2e-api-b-645aef-59b6d9d59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-645aef: serviceaccount "payload-processing-e2e-api-b-645aef" not found 46m Normal AddedInterface pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Add eth0 [10.134.0.159/23] from ovn-kubernetes 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Add eth0 [10.134.0.158/23] from ovn-kubernetes 46m Normal Pulled pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Container created 46m Normal Started pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Container started 46m Normal Pulled pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Container started 46m Normal Created pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Container created 46m Normal Pulled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Container created 46m Normal Started pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Container started 46m Warning Unhealthy pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Startup probe failed: Get "http://10.133.0.112:15021/healthz/ready": dial tcp 10.133.0.112:15021: connect: connection refused 46m Warning ProbeError pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Startup probe error: Get "http://10.133.0.112:15021/healthz/ready": dial tcp 10.133.0.112:15021: connect: connection refused... 46m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Readiness probe error: Get "http://10.134.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 46m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Readiness probe failed: Get "http://10.134.0.146:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 46m Warning ProbeError pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Startup probe error: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused... 46m Warning Unhealthy pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Startup probe failed: Get "http://10.132.0.111:15021/healthz/ready": dial tcp 10.132.0.111:15021: connect: connection refused 46m Normal ScalingReplicaSet deployment/e2e-api-a-288e01-openshift-default Scaled up replica set e2e-api-a-288e01-openshift-default-6bfb488db9 from 0 to 1 46m Normal SuccessfulCreate replicaset/e2e-api-a-288e01-openshift-default-6bfb488db9 Created pod: e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt 46m Normal AddedInterface pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Add eth0 [10.134.0.160/23] from ovn-kubernetes 46m Normal AddedLabel gateway/e2e-api-a-288e01 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-288e01 46m Normal Started pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Container started 46m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Container started 46m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Container created 46m Normal Created pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Container created 46m Normal Pulled pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-288e01 Scaled up replica set payload-pre-processing-e2e-api-a-288e01-dd6cdfb55 from 0 to 1 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-288e01 Scaled up replica set payload-processing-e2e-api-a-288e01-86b8bc958 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55 Created pod: payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55 Error creating: pods "payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-288e01: serviceaccount "payload-processing-e2e-api-a-288e01" not found 46m Warning FailedCreate replicaset/payload-processing-e2e-api-a-288e01-86b8bc958 Error creating: pods "payload-processing-e2e-api-a-288e01-86b8bc958-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-288e01: serviceaccount "payload-processing-e2e-api-a-288e01" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-288e01-86b8bc958 Created pod: payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl 46m Normal Pulled pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Add eth0 [10.133.0.121/23] from ovn-kubernetes 46m Normal Created pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Container created 46m Normal Started pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Container started 46m Normal AddedLabel gateway/e2e-inf-b-4225df Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-4225df 46m Normal Started pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Container started 46m Normal ScalingReplicaSet deployment/e2e-inf-b-4225df-openshift-default Scaled up replica set e2e-inf-b-4225df-openshift-default-99b8c8fff from 0 to 1 46m Normal Created pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Container created 46m Normal Pulled pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal SuccessfulCreate replicaset/e2e-inf-b-4225df-openshift-default-99b8c8fff Created pod: e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 46m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Add eth0 [10.134.0.162/23] from ovn-kubernetes 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-mt-ef844616-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-mt-ef844616-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal AddedInterface pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Add eth0 [10.133.0.122/23] from ovn-kubernetes 46m Warning Unhealthy pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Startup probe failed: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused 46m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Startup probe failed: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused 46m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Startup probe error: Get "http://10.134.0.146:15021/healthz/ready": dial tcp 10.134.0.146:15021: connect: connection refused... 46m Warning ProbeError pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Startup probe error: Get "http://10.134.0.148:15021/healthz/ready": dial tcp 10.134.0.148:15021: connect: connection refused... 46m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t Stopping container istio-proxy 46m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-qw55t 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 6; reason: All metrics below target 46m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled down replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 10 to 6 46m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled down replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 10 to 6 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 6; reason: All metrics below target 46m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 46m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj 46m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l 46m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Stopping container istio-proxy 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn 46m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-xv5vn Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-5nhpq Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-tl88l Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-j8dpb Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-np898 Stopping container istio-proxy 46m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-vqlvj Stopping container istio-proxy 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-4225df Scaled up replica set payload-processing-e2e-inf-b-4225df-6bd7847ff6 from 0 to 1 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-528d67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-528d67-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-4225df Scaled up replica set payload-pre-processing-e2e-inf-b-4225df-6ff59db555 from 0 to 1 46m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Readiness probe error: HTTP probe failed with statuscode: 503... 46m Normal AddedInterface pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Add eth0 [10.133.0.123/23] from ovn-kubernetes 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-4225df-6ff59db555 Created pod: payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-4225df-6ff59db555 Error creating: pods "payload-pre-processing-e2e-inf-b-4225df-6ff59db555-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-4225df: serviceaccount "payload-processing-e2e-inf-b-4225df" not found 46m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-4225df-6bd7847ff6 Error creating: pods "payload-processing-e2e-inf-b-4225df-6bd7847ff6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-4225df: serviceaccount "payload-processing-e2e-inf-b-4225df" not found 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-4225df-6bd7847ff6 Created pod: payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc 46m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6cbk Readiness probe failed: HTTP probe failed with statuscode: 503 46m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Add eth0 [10.134.0.163/23] from ovn-kubernetes 46m Normal Created pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Container created 46m Normal Started pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Container started 46m Normal Pulled pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Pulled pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Container created 46m Normal Started pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Container started 46m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 5; reason: All metrics below target 46m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 Stopping container istio-proxy 46m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-89p79 46m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled down replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 6 to 5 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-c613f5ea Scaled up replica set payload-pre-processing-e2e-mt-c613f5ea-55445d94bc from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc Created pod: payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g 46m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc Error creating: pods "payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c613f5ea: serviceaccount "payload-processing-e2e-mt-c613f5ea" not found 46m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-c613f5ea Scaled up replica set payload-processing-e2e-mt-c613f5ea-6c844f4556 from 0 to 1 46m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-c613f5ea-6c844f4556 Created pod: payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh 46m Warning FailedCreate replicaset/payload-processing-e2e-mt-c613f5ea-6c844f4556 Error creating: pods "payload-processing-e2e-mt-c613f5ea-6c844f4556-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c613f5ea: serviceaccount "payload-processing-e2e-mt-c613f5ea" not found 46m Normal Created pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Container created 46m Normal Started pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Container started 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-528d67-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal Pulled pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal AddedInterface pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Add eth0 [10.132.0.123/23] from ovn-kubernetes 46m Normal AddedInterface pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Add eth0 [10.132.0.124/23] from ovn-kubernetes 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-528d67-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 46m Normal Started pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Container started 46m Normal Created pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Container created 46m Normal Pulled pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 46m Normal Created pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Container created 46m Normal Pulled pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 46m Normal Started pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Container started 46m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 46m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 45m Warning ProbeError pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Startup probe error: Get "http://10.133.0.113:15021/healthz/ready": dial tcp 10.133.0.113:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Startup probe failed: Get "http://10.133.0.113:15021/healthz/ready": dial tcp 10.133.0.113:15021: connect: connection refused 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 6; reason: All metrics below target 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g 45m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-r479g Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 45m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-zck2c Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb 45m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-c6kd7 Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-fnfzb Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled down replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 10 to 6 45m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-s2l6v 45m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp Stopping container istio-proxy 45m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled down replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 10 to 8 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 8; reason: All metrics below target 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5xlvp 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 45m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-sggq6 Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-g6qmw 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n 45m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 6; reason: All metrics below target 45m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled down replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 10 to 6 45m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-xl8p4 Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Stopping container istio-proxy 45m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Readiness probe error: Get "http://10.132.0.64:15021/healthz/ready": read tcp 10.132.0.2:55294->10.132.0.64:15021: read: connection reset by peer... 45m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-t9g5n Readiness probe failed: Get "http://10.132.0.64:15021/healthz/ready": read tcp 10.132.0.2:55294->10.132.0.64:15021: read: connection reset by peer 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-ac96f3-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-ac96f3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Readiness probe error: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 Readiness probe failed: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled down replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 8 to 6 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 6; reason: All metrics below target 45m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft 45m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-r64tg Stopping container istio-proxy 45m Normal AddedInterface pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Add eth0 [10.132.0.128/23] from ovn-kubernetes 45m Normal SuccessfulCreate replicaset/e2e-api-b-3d55bd-openshift-default-6465745485 Created pod: e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz 45m Normal ScalingReplicaSet deployment/e2e-api-b-3d55bd-openshift-default Scaled up replica set e2e-api-b-3d55bd-openshift-default-6465745485 from 0 to 1 45m Normal AddedLabel gateway/e2e-api-b-3d55bd Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-3d55bd 45m Warning Unhealthy pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Readiness probe failed: Get "http://10.134.0.157:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Warning ProbeError pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Readiness probe error: Get "http://10.134.0.157:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-3d55bd Scaled up replica set payload-processing-e2e-api-b-3d55bd-59b8b9b87d from 0 to 1 45m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-3d55bd Scaled up replica set payload-pre-processing-e2e-api-b-3d55bd-fc4654867 from 0 to 1 45m Warning ProbeError pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Readiness probe error: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 45m Warning Unhealthy pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Readiness probe failed: Get "http://10.134.0.160:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 45m Warning FailedCreate replicaset/payload-processing-e2e-api-b-3d55bd-59b8b9b87d Error creating: pods "payload-processing-e2e-api-b-3d55bd-59b8b9b87d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-3d55bd: serviceaccount "payload-processing-e2e-api-b-3d55bd" not found 45m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-3d55bd-fc4654867 Created pod: payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 45m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-3d55bd-59b8b9b87d Created pod: payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 45m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-3d55bd-fc4654867 Error creating: pods "payload-pre-processing-e2e-api-b-3d55bd-fc4654867-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-3d55bd: serviceaccount "payload-processing-e2e-api-b-3d55bd" not found 45m Normal Created pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Container created 45m Normal Created pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Container created 45m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Add eth0 [10.132.0.129/23] from ovn-kubernetes 45m Normal Pulled pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal AddedInterface pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Add eth0 [10.134.0.165/23] from ovn-kubernetes 45m Normal Started pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Container started 45m Normal Pulled pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 45m Normal Started pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Container started 45m Normal Pulled pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Created pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Container created 45m Normal Started pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Container started 45m Normal Started pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Container started 45m Normal Created pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Container created 45m Normal Pulled pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal Killing pod/payload-processing-e2e-api-a-528d67-5998c8d466-84qzk Stopping container payload-processing 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-645aef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-645aef-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal Started pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Container started 45m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-r47sg Stopping container istio-proxy 45m Normal Created pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Container created 45m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled down replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 10 to 5 45m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 5; reason: All metrics below target 45m Normal Killing pod/payload-pre-processing-e2e-api-a-528d67-654d5c4795-h7bnj Stopping container payload-pre-processing 45m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 Stopping container istio-proxy 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv 45m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-s6khx Stopping container istio-proxy 45m Normal Pulled pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 45m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rgdt4 45m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v5t7m Stopping container istio-proxy 45m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-v8znv Stopping container istio-proxy 45m Warning Unhealthy pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Startup probe failed: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused 45m Warning ProbeError pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Startup probe error: Get "http://10.134.0.160:15021/healthz/ready": dial tcp 10.134.0.160:15021: connect: connection refused... 45m Warning ProbeError pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Startup probe error: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Startup probe failed: Get "http://10.134.0.157:15021/healthz/ready": dial tcp 10.134.0.157:15021: connect: connection refused 45m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-288e01-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-288e01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 45m Normal SuccessfulCreate replicaset/e2e-api-b-47af37-openshift-default-677f546d75 Created pod: e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 45m Normal ScalingReplicaSet deployment/e2e-api-b-47af37-openshift-default Scaled up replica set e2e-api-b-47af37-openshift-default-677f546d75 from 0 to 1 45m Normal AddedLabel gateway/e2e-api-b-47af37 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-47af37 45m Normal AddedInterface pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Add eth0 [10.132.0.130/23] from ovn-kubernetes 45m Warning ProbeError pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Startup probe error: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused... 45m Warning Unhealthy pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Startup probe failed: Get "http://10.134.0.156:15021/healthz/ready": dial tcp 10.134.0.156:15021: connect: connection refused 45m Normal Killing pod/payload-pre-processing-e2e-mt-ef844616-7b44b7cd87-fj2dx Stopping container payload-pre-processing 45m Normal Killing pod/payload-processing-e2e-mt-ef844616-5cb9d44b8-sb45v Stopping container payload-processing 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-47af37 Scaled up replica set payload-processing-e2e-api-b-47af37-68cd9f46b4 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-47af37-68cd9f46b4 Created pod: payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw 44m Warning FailedCreate replicaset/payload-processing-e2e-api-b-47af37-68cd9f46b4 Error creating: pods "payload-processing-e2e-api-b-47af37-68cd9f46b4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-47af37: serviceaccount "payload-processing-e2e-api-b-47af37" not found 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-47af37 Scaled up replica set payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5 from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5 Created pod: payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5 Error creating: pods "payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-47af37: serviceaccount "payload-processing-e2e-api-b-47af37" not found 44m Normal Created pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Container created 44m Normal Started pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Container started 44m Normal AddedInterface pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Add eth0 [10.132.0.132/23] from ovn-kubernetes 44m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Add eth0 [10.132.0.131/23] from ovn-kubernetes 44m Normal Started pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Container started 44m Normal Pulled pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Pulled pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Container created 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-543d08-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-543d08-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-645aef-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-543d08-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-645aef-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Stopping container istio-proxy 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-543d08-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 5; reason: All metrics below target 44m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c Stopping container istio-proxy 44m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 Stopping container istio-proxy 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-xcz58 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbj2c 44m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled down replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 10 to 5 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-pr676 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd 44m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd 44m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Readiness probe failed: HTTP probe failed with statuscode: 503 44m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-rwntd Readiness probe error: HTTP probe failed with statuscode: 503... 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-288e01-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-288e01-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Normal Killing pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc Stopping container istio-proxy 44m Normal Killing pod/payload-processing-e2e-mt-c613f5ea-6c844f4556-fvjwh Stopping container payload-processing 44m Normal Killing pod/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-ktl4g Stopping container payload-pre-processing 44m Normal AddedLabel gateway/e2e-api-a-b0814c Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-b0814c 44m Normal ScalingReplicaSet deployment/e2e-api-a-b0814c-openshift-default Scaled up replica set e2e-api-a-b0814c-openshift-default-768f4b658f from 0 to 1 44m Normal AddedInterface pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Add eth0 [10.132.0.136/23] from ovn-kubernetes 44m Normal SuccessfulCreate replicaset/e2e-api-a-b0814c-openshift-default-768f4b658f Created pod: e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 44m Normal ScalingReplicaSet deployment/e2e-mt-c613f5ea-openshift-default Scaled up replica set e2e-mt-c613f5ea-openshift-default-55cfff77f7 from 2 to 4 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 44m Normal Started pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Container started 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Add eth0 [10.132.0.137/23] from ovn-kubernetes 44m Normal Created pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Container created 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Container started 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Container created 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Add eth0 [10.132.0.138/23] from ovn-kubernetes 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Container created 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Container started 44m Normal Pulled pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-b0814c Scaled up replica set payload-pre-processing-e2e-api-a-b0814c-59c85fb755 from 0 to 1 44m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-b0814c-59c85fb755 Error creating: pods "payload-pre-processing-e2e-api-a-b0814c-59c85fb755-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-b0814c: serviceaccount "payload-processing-e2e-api-a-b0814c" not found 44m Warning FailedCreate replicaset/payload-processing-e2e-api-a-b0814c-75cccd9f6b Error creating: pods "payload-processing-e2e-api-a-b0814c-75cccd9f6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-b0814c: serviceaccount "payload-processing-e2e-api-a-b0814c" not found 44m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-b0814c-75cccd9f6b Created pod: payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp 44m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-b0814c Scaled up replica set payload-processing-e2e-api-a-b0814c-75cccd9f6b from 0 to 1 44m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-b0814c-59c85fb755 Created pod: payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 44m Normal Pulled pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Created pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Container created 44m Normal Created pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Container created 44m Normal Started pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Container started 44m Normal Pulled pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 44m Normal Started pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Container started 44m Normal AddedInterface pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Add eth0 [10.132.0.139/23] from ovn-kubernetes 44m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Add eth0 [10.132.0.140/23] from ovn-kubernetes 44m Warning ProbeError pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Startup probe error: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Startup probe failed: Get "http://10.133.0.122:15021/healthz/ready": dial tcp 10.133.0.122:15021: connect: connection refused 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Startup probe error: Get "http://10.132.0.138:15021/healthz/ready": dial tcp 10.132.0.138:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Startup probe failed: Get "http://10.132.0.138:15021/healthz/ready": dial tcp 10.132.0.138:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Startup probe failed: Get "http://10.132.0.137:15021/healthz/ready": dial tcp 10.132.0.137:15021: connect: connection refused 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Startup probe error: Get "http://10.132.0.137:15021/healthz/ready": dial tcp 10.132.0.137:15021: connect: connection refused... 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Container created 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk 44m Normal ScalingReplicaSet deployment/e2e-mt-c613f5ea-openshift-default Scaled up replica set e2e-mt-c613f5ea-openshift-default-55cfff77f7 from 4 to 8 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Container created 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Add eth0 [10.134.0.169/23] from ovn-kubernetes 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Container started 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Add eth0 [10.134.0.170/23] from ovn-kubernetes 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Container started 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Add eth0 [10.133.0.124/23] from ovn-kubernetes 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Container created 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Container started 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Container created 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Add eth0 [10.132.0.143/23] from ovn-kubernetes 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Container started 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-4225df-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-4225df-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Startup probe error: Get "http://10.132.0.143:15021/healthz/ready": dial tcp 10.132.0.143:15021: connect: connection refused... 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Startup probe error: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Startup probe failed: Get "http://10.133.0.124:15021/healthz/ready": dial tcp 10.133.0.124:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Startup probe failed: Get "http://10.132.0.143:15021/healthz/ready": dial tcp 10.132.0.143:15021: connect: connection refused 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Startup probe error: Get "http://10.134.0.170:15021/healthz/ready": dial tcp 10.134.0.170:15021: connect: connection refused... 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Startup probe failed: Get "http://10.134.0.170:15021/healthz/ready": dial tcp 10.134.0.170:15021: connect: connection refused 44m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Startup probe failed: Get "http://10.134.0.169:15021/healthz/ready": dial tcp 10.134.0.169:15021: connect: connection refused 44m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Startup probe error: Get "http://10.134.0.169:15021/healthz/ready": dial tcp 10.134.0.169:15021: connect: connection refused... 44m Normal Killing pod/e2e-api-a-528d67-openshift-default-5c45549d45-bn8zm Stopping container istio-proxy 44m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-b0814c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-b0814c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Add eth0 [10.132.0.144/23] from ovn-kubernetes 44m Normal SuccessfulRescale horizontalpodautoscaler/e2e-mt-c613f5ea-openshift-default New size: 10; reason: 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 (combined from similar events): Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q 44m Normal SuccessfulCreate replicaset/e2e-mt-c613f5ea-openshift-default-55cfff77f7 Created pod: e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk 44m Normal AddedInterface pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Add eth0 [10.133.0.125/23] from ovn-kubernetes 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Container started 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Container created 44m Normal Created pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Container created 44m Normal Started pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Container started 44m Normal Pulled pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 44m Normal ScalingReplicaSet deployment/e2e-mt-c613f5ea-openshift-default Scaled up replica set e2e-mt-c613f5ea-openshift-default-55cfff77f7 from 8 to 10 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-4225df-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-4225df-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Killing pod/payload-processing-e2e-api-b-645aef-5b789579fd-2xhbs Stopping container payload-processing 43m Normal Killing pod/payload-pre-processing-e2e-api-b-645aef-59b6d9d59-pbrfk Stopping container payload-pre-processing 43m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Startup probe error: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Startup probe failed: Get "http://10.132.0.144:15021/healthz/ready": dial tcp 10.132.0.144:15021: connect: connection refused 43m Warning ProbeError pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Startup probe error: Get "http://10.133.0.125:15021/healthz/ready": dial tcp 10.133.0.125:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Startup probe failed: Get "http://10.133.0.125:15021/healthz/ready": dial tcp 10.133.0.125:15021: connect: connection refused 43m Normal Pulled pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Created pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Container created 43m Normal Started pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Container started 43m Normal Killing pod/payload-processing-e2e-api-a-543d08-6787968b78-jpdvp Stopping container payload-processing 43m Normal Killing pod/payload-pre-processing-e2e-api-a-543d08-7844bd78d7-9m5dh Stopping container payload-pre-processing 43m Warning ProbeError pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Startup probe error: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Startup probe failed: Get "http://10.132.0.128:15021/healthz/ready": dial tcp 10.132.0.128:15021: connect: connection refused 43m Normal ScalingReplicaSet deployment/e2e-inf-b-4225df-openshift-default Scaled up replica set e2e-inf-b-4225df-openshift-default-99b8c8fff from 1 to 2 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-4225df-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 43m Normal SuccessfulCreate replicaset/e2e-inf-b-4225df-openshift-default-99b8c8fff Created pod: e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 43m Normal Created pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Container created 43m Normal AddedInterface pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Add eth0 [10.134.0.174/23] from ovn-kubernetes 43m Normal Pulled pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal Started pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Container started 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-3d55bd-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-3d55bd-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 43m Warning ProbeError pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Startup probe error: Get "http://10.134.0.174:15021/healthz/ready": dial tcp 10.134.0.174:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Startup probe failed: Get "http://10.134.0.174:15021/healthz/ready": dial tcp 10.134.0.174:15021: connect: connection refused 43m Normal Started pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Container started 43m Normal Created pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Container created 43m Normal Pulled pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal ScalingReplicaSet deployment/e2e-inf-a-ac96f3-openshift-default Scaled up replica set e2e-inf-a-ac96f3-openshift-default-86c6f75967 from 1 to 2 43m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-ac96f3-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 43m Normal SuccessfulCreate replicaset/e2e-inf-a-ac96f3-openshift-default-86c6f75967 Created pod: e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-47af37-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-47af37-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal Pulled pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Add eth0 [10.134.0.175/23] from ovn-kubernetes 43m Normal Started pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Container started 43m Normal Created pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Container created 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-2lshk Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k Stopping container istio-proxy 43m Normal Killing pod/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl Stopping container payload-processing 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-hxp9t Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-rbvhv Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-w48nm Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n8m8q Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-n5k8s Stopping container istio-proxy 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-xnp92 Stopping container istio-proxy 43m Normal Killing pod/payload-pre-processing-e2e-api-a-288e01-dd6cdfb55-flr4p Stopping container payload-pre-processing 43m Normal Killing pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk Stopping container istio-proxy 43m Warning ProbeError pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Startup probe error: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused... 43m Warning Unhealthy pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Startup probe failed: Get "http://10.132.0.130:15021/healthz/ready": dial tcp 10.132.0.130:15021: connect: connection refused 43m Warning Unhealthy pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Startup probe failed: Get "http://10.134.0.175:15021/healthz/ready": dial tcp 10.134.0.175:15021: connect: connection refused 43m Warning ProbeError pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Startup probe error: Get "http://10.134.0.175:15021/healthz/ready": dial tcp 10.134.0.175:15021: connect: connection refused... 43m Normal AddedLabel gateway/e2e-inf-a-c7f962 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-c7f962 43m Normal ScalingReplicaSet deployment/e2e-inf-a-c7f962-openshift-default Scaled up replica set e2e-inf-a-c7f962-openshift-default-858f5c9bf from 0 to 1 43m Normal SuccessfulCreate replicaset/e2e-inf-a-c7f962-openshift-default-858f5c9bf Created pod: e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs 43m Normal AddedInterface pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Add eth0 [10.132.0.146/23] from ovn-kubernetes 43m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-3d55bd-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-3d55bd-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 43m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-c7f962-6886679566 Created pod: payload-processing-e2e-inf-a-c7f962-6886679566-84khk 43m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7 Error creating: pods "payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-c7f962: serviceaccount "payload-processing-e2e-inf-a-c7f962" not found 43m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-c7f962-6886679566 Error creating: pods "payload-processing-e2e-inf-a-c7f962-6886679566-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-c7f962: serviceaccount "payload-processing-e2e-inf-a-c7f962" not found 43m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7 Created pod: payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs 43m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-c7f962 Scaled up replica set payload-pre-processing-e2e-inf-a-c7f962-85df567dd7 from 0 to 1 43m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-c7f962 Scaled up replica set payload-processing-e2e-inf-a-c7f962-6886679566 from 0 to 1 43m Normal Created pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Container created 43m Normal Started pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Container started 43m Normal AddedInterface pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Add eth0 [10.132.0.147/23] from ovn-kubernetes 43m Normal Pulled pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Add eth0 [10.134.0.177/23] from ovn-kubernetes 43m Normal Started pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Container started 43m Normal Created pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Container created 43m Normal Pulled pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 43m Normal AddedLabel gateway/e2e-inf-a-1f8eb4 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-1f8eb4 42m Normal ScalingReplicaSet deployment/e2e-inf-a-1f8eb4-openshift-default Scaled up replica set e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 from 0 to 1 42m Normal SuccessfulCreate replicaset/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 Created pod: e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq 42m Normal AddedInterface pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Add eth0 [10.134.0.178/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Readiness probe error: Get "http://10.132.0.136:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Warning Unhealthy pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Readiness probe failed: Get "http://10.132.0.136:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 42m Normal Pulled pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 42m Normal Created pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Container created 42m Normal Started pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Container started 42m Normal AddedLabel gateway/e2e-api-b-c3486b Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-c3486b 42m Normal SuccessfulCreate replicaset/e2e-api-b-c3486b-openshift-default-7684fb8cf Created pod: e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk 42m Normal ScalingReplicaSet deployment/e2e-api-b-c3486b-openshift-default Scaled up replica set e2e-api-b-c3486b-openshift-default-7684fb8cf from 0 to 1 42m Normal AddedInterface pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Add eth0 [10.134.0.179/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-1f8eb4-654c4687f6 Created pod: payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-1f8eb4 Scaled up replica set payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59 Created pod: payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59 Error creating: pods "payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1f8eb4: serviceaccount "payload-processing-e2e-inf-a-1f8eb4" not found 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-1f8eb4-654c4687f6 Error creating: pods "payload-processing-e2e-inf-a-1f8eb4-654c4687f6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-1f8eb4: serviceaccount "payload-processing-e2e-inf-a-1f8eb4" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-1f8eb4 Scaled up replica set payload-processing-e2e-inf-a-1f8eb4-654c4687f6 from 0 to 1 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Add eth0 [10.134.0.180/23] from ovn-kubernetes 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Created pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Container created 42m Normal Started pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Container started 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Add eth0 [10.132.0.148/23] from ovn-kubernetes 42m Normal Created pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Container created 42m Normal Started pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Container started 42m Normal Pulled pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Warning Unhealthy pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Startup probe failed: Get "http://10.132.0.136:15021/healthz/ready": dial tcp 10.132.0.136:15021: connect: connection refused 42m Warning ProbeError pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Startup probe error: Get "http://10.132.0.136:15021/healthz/ready": dial tcp 10.132.0.136:15021: connect: connection refused... 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-c3486b Scaled up replica set payload-processing-e2e-api-b-c3486b-67cf79f596 from 0 to 1 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff Created pod: payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-c3486b-67cf79f596 Created pod: payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc 42m Warning FailedCreate replicaset/payload-processing-e2e-api-b-c3486b-67cf79f596 Error creating: pods "payload-processing-e2e-api-b-c3486b-67cf79f596-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-c3486b: serviceaccount "payload-processing-e2e-api-b-c3486b" not found 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-c3486b Scaled up replica set payload-pre-processing-e2e-api-b-c3486b-585f5c44ff from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff Error creating: pods "payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-c3486b: serviceaccount "payload-processing-e2e-api-b-c3486b" not found 42m Normal Pulled pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Add eth0 [10.132.0.149/23] from ovn-kubernetes 42m Normal Started pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Container started 42m Normal Started pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Container started 42m Normal Created pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Container created 42m Normal AddedInterface pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Add eth0 [10.132.0.150/23] from ovn-kubernetes 42m Normal Created pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Container created 42m Normal Pulled pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-47af37-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-47af37-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-a-b0814c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-a-b0814c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 42m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl Stopping container istio-proxy 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 5; reason: All metrics below target 42m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d8lbl 42m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default Scaled down replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 6 to 5 42m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 42m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 5; reason: All metrics below target 42m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Stopping container istio-proxy 42m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled down replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 6 to 5 42m Normal Killing pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Stopping container istio-proxy 42m Normal Killing pod/e2e-api-b-645aef-openshift-default-7fd469db85-tdz8g Stopping container istio-proxy 42m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Readiness probe failed: Get "http://10.133.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 42m Normal AddedLabel gateway/e2e-inf-a-010013 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-010013 42m Normal ScalingReplicaSet deployment/e2e-inf-a-010013-openshift-default Scaled up replica set e2e-inf-a-010013-openshift-default-59977f9669 from 0 to 1 42m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 Readiness probe error: Get "http://10.133.0.80:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 42m Normal SuccessfulCreate replicaset/e2e-inf-a-010013-openshift-default-59977f9669 Created pod: e2e-inf-a-010013-openshift-default-59977f9669-scchv 42m Warning Unhealthy pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Readiness probe failed: HTTP probe failed with statuscode: 503 42m Normal AddedInterface pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Add eth0 [10.134.0.182/23] from ovn-kubernetes 42m Warning ProbeError pod/e2e-api-a-543d08-openshift-default-f45f9c965-597sx Readiness probe error: HTTP probe failed with statuscode: 503... 42m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-010013 Scaled up replica set payload-pre-processing-e2e-inf-a-010013-744897b695 from 0 to 1 42m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-010013-65dc787d94 Error creating: pods "payload-processing-e2e-inf-a-010013-65dc787d94-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-010013: serviceaccount "payload-processing-e2e-inf-a-010013" not found 42m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-010013 Scaled up replica set payload-processing-e2e-inf-a-010013-65dc787d94 from 0 to 1 42m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-010013-744897b695 Error creating: pods "payload-pre-processing-e2e-inf-a-010013-744897b695-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-010013: serviceaccount "payload-processing-e2e-inf-a-010013" not found 42m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Add eth0 [10.132.0.152/23] from ovn-kubernetes 42m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-010013-65dc787d94 Created pod: payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn 42m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-010013-744897b695 Created pod: payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t 42m Normal Created pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Container created 42m Normal Pulled pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal Started pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Container started 42m Normal Started pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Container started 42m Normal Pulled pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 42m Normal AddedInterface pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Add eth0 [10.132.0.153/23] from ovn-kubernetes 42m Normal Created pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Container created 42m Normal Killing pod/payload-pre-processing-e2e-api-b-3d55bd-fc4654867-q5cm2 Stopping container payload-pre-processing 42m Normal Killing pod/payload-processing-e2e-api-b-3d55bd-59b8b9b87d-tjtj8 Stopping container payload-processing 42m Normal Killing pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Stopping container istio-proxy 42m Warning Unhealthy pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Readiness probe failed: HTTP probe failed with statuscode: 503 42m Warning ProbeError pod/e2e-api-a-288e01-openshift-default-6bfb488db9-d9szt Readiness probe error: HTTP probe failed with statuscode: 503... 41m Normal Killing pod/payload-pre-processing-e2e-api-b-47af37-75b4cc6ff5-h2f28 Stopping container payload-pre-processing 41m Normal Killing pod/payload-processing-e2e-api-b-47af37-68cd9f46b4-ddgjw Stopping container payload-processing 41m Normal Pulled pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Normal Started pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Container started 41m Normal Created pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Container created 41m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 5; reason: All metrics below target 41m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled down replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 6 to 5 41m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-c7f962-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-c7f962-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 41m Warning ProbeError pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Startup probe error: Get "http://10.132.0.146:15021/healthz/ready": dial tcp 10.132.0.146:15021: connect: connection refused... 41m Warning Unhealthy pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Startup probe failed: Get "http://10.132.0.146:15021/healthz/ready": dial tcp 10.132.0.146:15021: connect: connection refused 41m Normal AddedLabel gateway/e2e-inf-b-453511 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-453511 41m Normal ScalingReplicaSet deployment/e2e-inf-b-453511-openshift-default Scaled up replica set e2e-inf-b-453511-openshift-default-9db66fcf5 from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-b-453511-openshift-default-9db66fcf5 Created pod: e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk 41m Normal AddedInterface pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Add eth0 [10.134.0.187/23] from ovn-kubernetes 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-453511-6778fc7699 Created pod: payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-453511 Scaled up replica set payload-pre-processing-e2e-inf-b-453511-6778fc7699 from 0 to 1 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-453511-6778fc7699 Error creating: pods "payload-pre-processing-e2e-inf-b-453511-6778fc7699-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-453511: serviceaccount "payload-processing-e2e-inf-b-453511" not found 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-453511 Scaled up replica set payload-processing-e2e-inf-b-453511-5d55648d from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-453511-5d55648d Created pod: payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-453511-5d55648d Error creating: pods "payload-processing-e2e-inf-b-453511-5d55648d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-453511: serviceaccount "payload-processing-e2e-inf-b-453511" not found 41m Normal Killing pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Stopping container payload-processing 41m Normal Started pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Container started 41m Normal AddedInterface pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Add eth0 [10.134.0.188/23] from ovn-kubernetes 41m Normal Killing pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Stopping container payload-pre-processing 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Add eth0 [10.134.0.189/23] from ovn-kubernetes 41m Normal Pulled pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Container created 41m Normal Pulled pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Started pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Container started 41m Normal Created pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Container created 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default New size: 4; reason: All metrics below target 41m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm Stopping container istio-proxy 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 3; reason: All metrics below target 41m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 41m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr 41m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm 41m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default Scaled down replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 6 to 5 41m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 5; reason: All metrics below target 41m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Stopping container istio-proxy 41m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk 41m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled down replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 5 to 3 41m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr Stopping container istio-proxy 41m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Readiness probe failed: HTTP probe failed with statuscode: 503 41m Normal AddedLabel gateway/e2e-inf-b-36fd3c Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-36fd3c 41m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-bgzfk Readiness probe error: HTTP probe failed with statuscode: 503... 41m Normal ScalingReplicaSet deployment/e2e-inf-b-36fd3c-openshift-default Scaled up replica set e2e-inf-b-36fd3c-openshift-default-cc6c6c87f from 0 to 1 41m Normal SuccessfulCreate replicaset/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f Created pod: e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn 41m Normal AddedInterface pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Add eth0 [10.134.0.190/23] from ovn-kubernetes 41m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Readiness probe error: HTTP probe failed with statuscode: 503... 41m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 Readiness probe failed: HTTP probe failed with statuscode: 503 41m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5 Created pod: payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc 41m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-36fd3c Scaled up replica set payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5 from 0 to 1 41m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-36fd3c Scaled up replica set payload-processing-e2e-inf-b-36fd3c-7895ccc8dd from 0 to 1 41m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd Created pod: payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf 41m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd Error creating: pods "payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-36fd3c: serviceaccount "payload-processing-e2e-inf-b-36fd3c" not found 41m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5 Error creating: pods "payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-36fd3c: serviceaccount "payload-processing-e2e-inf-b-36fd3c" not found 41m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Add eth0 [10.132.0.157/23] from ovn-kubernetes 41m Normal Started pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Container started 41m Normal Created pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Container created 41m Normal Pulled pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Pulled pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 41m Normal Created pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Container created 41m Normal AddedInterface pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Add eth0 [10.134.0.192/23] from ovn-kubernetes 41m Normal Started pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Container started 41m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-ac96f3-f75f69799-vb7ch Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Warning Unhealthy pod/payload-processing-e2e-inf-a-ac96f3-55cf8f4776-g25zl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 41m Normal Killing pod/payload-pre-processing-e2e-api-a-b0814c-59c85fb755-g74w7 Stopping container payload-pre-processing 41m Normal Killing pod/payload-processing-e2e-api-a-b0814c-75cccd9f6b-n4mwp Stopping container payload-processing 41m Normal Pulled pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 41m Warning ProbeError pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Readiness probe error: Get "http://10.134.0.182:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 41m Warning Unhealthy pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Readiness probe failed: Get "http://10.134.0.182:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 40m Normal Created pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Container created 40m Normal Pulled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal ScalingReplicaSet deployment/e2e-inf-a-1f8eb4-openshift-default Scaled up replica set e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 from 1 to 4 40m Normal SuccessfulCreate replicaset/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 Created pod: e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 40m Normal SuccessfulCreate replicaset/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 Created pod: e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv 40m Normal SuccessfulCreate replicaset/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75 Created pod: e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz 40m Normal Started pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Container started 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-1f8eb4-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 40m Normal Pulled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Started pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Container started 40m Normal Started pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Container started 40m Normal Created pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Container created 40m Normal Created pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Container created 40m Normal Pulled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Add eth0 [10.132.0.158/23] from ovn-kubernetes 40m Normal AddedInterface pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Add eth0 [10.133.0.126/23] from ovn-kubernetes 40m Normal Started pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Container started 40m Normal AddedInterface pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Add eth0 [10.134.0.193/23] from ovn-kubernetes 40m Normal Pulled pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Created pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Container created 40m Normal Created pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Container created 40m Normal Started pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Container started 40m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default New size: 4; reason: All metrics below target 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 4; reason: All metrics below target 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 4; reason: All metrics below target 40m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf 40m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled down replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 5 to 4 40m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-578tf Stopping container istio-proxy 40m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled down replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 5 to 4 40m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-c7f962-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-c7f962-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Stopping container istio-proxy 40m Warning Unhealthy pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Startup probe failed: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Startup probe error: Get "http://10.133.0.126:15021/healthz/ready": dial tcp 10.133.0.126:15021: connect: connection refused... 40m Warning ProbeError pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Startup probe error: Get "http://10.132.0.158:15021/healthz/ready": dial tcp 10.132.0.158:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Startup probe failed: Get "http://10.132.0.158:15021/healthz/ready": dial tcp 10.132.0.158:15021: connect: connection refused 40m Warning Unhealthy pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Startup probe failed: Get "http://10.134.0.182:15021/healthz/ready": dial tcp 10.134.0.182:15021: connect: connection refused 40m Warning ProbeError pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Startup probe error: Get "http://10.134.0.182:15021/healthz/ready": dial tcp 10.134.0.182:15021: connect: connection refused... 40m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal Killing pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Stopping container istio-proxy 40m Warning ProbeError pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Readiness probe error: HTTP probe failed with statuscode: 503... 40m Normal AddedLabel gateway/e2e-inf-a-ccafe0 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-ccafe0 40m Warning ProbeError pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Startup probe error: Get "http://10.134.0.178:15021/healthz/ready": dial tcp 10.134.0.178:15021: connect: connection refused... 40m Normal ScalingReplicaSet deployment/e2e-inf-a-ccafe0-openshift-default Scaled up replica set e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9 from 0 to 1 40m Warning Unhealthy pod/e2e-api-b-3d55bd-openshift-default-6465745485-wz9dz Readiness probe failed: HTTP probe failed with statuscode: 503 40m Normal SuccessfulCreate replicaset/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9 Created pod: e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f 40m Warning Unhealthy pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Startup probe failed: Get "http://10.134.0.178:15021/healthz/ready": dial tcp 10.134.0.178:15021: connect: connection refused 40m Normal AddedInterface pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Add eth0 [10.132.0.160/23] from ovn-kubernetes 40m Warning ProbeError pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Startup probe error: Get "http://10.134.0.193:15021/healthz/ready": dial tcp 10.134.0.193:15021: connect: connection refused... 40m Warning Unhealthy pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Startup probe failed: Get "http://10.134.0.193:15021/healthz/ready": dial tcp 10.134.0.193:15021: connect: connection refused 40m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Readiness probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused 40m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-8br78 Readiness probe error: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused... 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-010013-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-010013-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Normal Killing pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 Stopping container istio-proxy 40m Normal Killing pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-bhqcc Stopping container istio-proxy 40m Normal AddedLabel gateway/e2e-inf-b-8afb26 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-8afb26 40m Normal SuccessfulCreate replicaset/e2e-inf-b-8afb26-openshift-default-6d8b784f7f Created pod: e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 40m Normal AddedInterface pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Add eth0 [10.134.0.194/23] from ovn-kubernetes 40m Normal ScalingReplicaSet deployment/e2e-inf-b-8afb26-openshift-default Scaled up replica set e2e-inf-b-8afb26-openshift-default-6d8b784f7f from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-ccafe0 Scaled up replica set payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447 from 0 to 1 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996 Error creating: pods "payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-ccafe0: serviceaccount "payload-processing-e2e-inf-a-ccafe0" not found 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1f8eb4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1f8eb4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447 Error creating: pods "payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-ccafe0: serviceaccount "payload-processing-e2e-inf-a-ccafe0" not found 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447 Created pod: payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996 Created pod: payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-ccafe0 Scaled up replica set payload-processing-e2e-inf-a-ccafe0-7bbc9b8996 from 0 to 1 40m Normal Started pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Container started 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Add eth0 [10.132.0.161/23] from ovn-kubernetes 40m Normal Started pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Container started 40m Normal AddedInterface pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Add eth0 [10.132.0.162/23] from ovn-kubernetes 40m Normal Pulled pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Container created 40m Normal Created pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Container created 40m Normal Pulled pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-8afb26 Scaled up replica set payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879 from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-8afb26 Scaled up replica set payload-processing-e2e-inf-b-8afb26-864b7b9f48 from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879 Created pod: payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879 Error creating: pods "payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-8afb26: serviceaccount "payload-processing-e2e-inf-b-8afb26" not found 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-8afb26-864b7b9f48 Created pod: payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Add eth0 [10.134.0.196/23] from ovn-kubernetes 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-8afb26-864b7b9f48 Error creating: pods "payload-processing-e2e-inf-b-8afb26-864b7b9f48-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-8afb26: serviceaccount "payload-processing-e2e-inf-b-8afb26" not found 40m Normal Pulled pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Container created 40m Normal Started pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Container started 40m Normal Started pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Container started 40m Normal Created pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Container created 40m Normal Pulled pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Add eth0 [10.134.0.197/23] from ovn-kubernetes 40m Normal Killing pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Stopping container payload-pre-processing 40m Normal Killing pod/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc Stopping container payload-processing 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-010013-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-010013-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Normal Killing pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Stopping container istio-proxy 40m Normal AddedLabel gateway/e2e-inf-a-356cae Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-356cae 40m Normal ScalingReplicaSet deployment/e2e-inf-a-356cae-openshift-default Scaled up replica set e2e-inf-a-356cae-openshift-default-5bd47d6d5 from 0 to 1 40m Normal SuccessfulCreate replicaset/e2e-inf-a-356cae-openshift-default-5bd47d6d5 Created pod: e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w 40m Normal AddedInterface pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Add eth0 [10.133.0.127/23] from ovn-kubernetes 40m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-1f8eb4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-1f8eb4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 40m Warning ProbeError pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Readiness probe error: HTTP probe failed with statuscode: 503... 40m Warning Unhealthy pod/e2e-api-b-47af37-openshift-default-677f546d75-h4nn4 Readiness probe failed: HTTP probe failed with statuscode: 503 40m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-356cae-795777684b Error creating: pods "payload-processing-e2e-inf-a-356cae-795777684b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-356cae: serviceaccount "payload-processing-e2e-inf-a-356cae" not found 40m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-356cae-795777684b Created pod: payload-processing-e2e-inf-a-356cae-795777684b-ftbfw 40m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-356cae-7765997747 Error creating: pods "payload-pre-processing-e2e-inf-a-356cae-7765997747-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-356cae: serviceaccount "payload-processing-e2e-inf-a-356cae" not found 40m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-356cae Scaled up replica set payload-pre-processing-e2e-inf-a-356cae-7765997747 from 0 to 1 40m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-356cae Scaled up replica set payload-processing-e2e-inf-a-356cae-795777684b from 0 to 1 40m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-356cae-7765997747 Created pod: payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg 40m Normal Pulled pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal Created pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Container created 40m Normal Started pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Container started 40m Normal Pulled pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 40m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Add eth0 [10.132.0.165/23] from ovn-kubernetes 40m Normal AddedInterface pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Add eth0 [10.132.0.164/23] from ovn-kubernetes 40m Normal Created pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Container created 40m Normal Started pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Container started 40m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-4225df-6ff59db555-rf2ft Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 40m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Stopping container istio-proxy 40m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 3; reason: All metrics below target 40m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q Stopping container istio-proxy 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 40m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled down replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 5 to 3 40m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q 40m Warning Unhealthy pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Readiness probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 40m Normal Killing pod/e2e-api-a-b0814c-openshift-default-768f4b658f-6z7p4 Stopping container istio-proxy 40m Warning ProbeError pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 Readiness probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 40m Normal Pulled pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 40m Normal Started pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Container started 40m Normal Created pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Container created 39m Warning Unhealthy pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Startup probe failed: Get "http://10.134.0.187:15021/healthz/ready": dial tcp 10.134.0.187:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Startup probe error: Get "http://10.134.0.187:15021/healthz/ready": dial tcp 10.134.0.187:15021: connect: connection refused... 39m Normal Killing pod/payload-processing-e2e-api-b-c3486b-67cf79f596-2kzcc Stopping container payload-processing 39m Normal Killing pod/payload-pre-processing-e2e-api-b-c3486b-585f5c44ff-wxpnk Stopping container payload-pre-processing 39m Normal Killing pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 Stopping container istio-proxy 39m Normal Killing pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-qc478 Stopping container istio-proxy 39m Normal ScalingReplicaSet deployment/e2e-shared-a-w6-2f6fed-openshift-default Scaled up replica set e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89 from 0 to 1 39m Normal Started pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Container started 39m Normal AddedLabel gateway/e2e-shared-a-w6-2f6fed Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w6-2f6fed 39m Normal Pulled pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal SuccessfulCreate replicaset/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89 Created pod: e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt 39m Normal Created pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Container created 39m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Stopping container istio-proxy 39m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 39m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 4; reason: All metrics below target 39m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled down replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 5 to 4 39m Normal AddedInterface pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Add eth0 [10.132.0.168/23] from ovn-kubernetes 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-453511-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-453511-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46 Error creating: pods "payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed: serviceaccount "payload-processing-e2e-shared-a-w6-2f6fed" not found 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46 Created pod: payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w6-2f6fed Scaled up replica set payload-processing-e2e-shared-a-w6-2f6fed-76f888db46 from 0 to 1 39m Warning ProbeError pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Startup probe error: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused... 39m Warning Unhealthy pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Startup probe failed: Get "http://10.134.0.179:15021/healthz/ready": dial tcp 10.134.0.179:15021: connect: connection refused 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f Created pod: payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f Error creating: pods "payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w6-2f6fed: serviceaccount "payload-processing-e2e-shared-a-w6-2f6fed" not found 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w6-2f6fed Scaled up replica set payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f from 0 to 1 39m Normal Created pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Container created 39m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Readiness probe error: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 39m Normal Started pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Container started 39m Normal Created pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Container created 39m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Add eth0 [10.132.0.170/23] from ovn-kubernetes 39m Normal AddedInterface pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Add eth0 [10.132.0.169/23] from ovn-kubernetes 39m Normal Pulled pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Started pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Container started 39m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sjjr8 Readiness probe failed: Get "http://10.134.0.62:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-c3486b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-c3486b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-453511-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-453511-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Normal Started pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Container started 39m Normal Created pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Container created 39m Normal Pulled pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Pulled pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 39m Normal Created pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Container created 39m Normal Started pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Container started 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-8afb26-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-8afb26-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning Unhealthy pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Startup probe failed: Get "http://10.134.0.194:15021/healthz/ready": dial tcp 10.134.0.194:15021: connect: connection refused 39m Warning ProbeError pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Startup probe error: Get "http://10.134.0.194:15021/healthz/ready": dial tcp 10.134.0.194:15021: connect: connection refused... 39m Normal AddedLabel gateway/e2e-inf-b-2a8f18 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-2a8f18 39m Normal ScalingReplicaSet deployment/e2e-inf-b-2a8f18-openshift-default Scaled up replica set e2e-inf-b-2a8f18-openshift-default-74685fbccc from 0 to 1 39m Normal SuccessfulCreate replicaset/e2e-inf-b-2a8f18-openshift-default-74685fbccc Created pod: e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p 39m Warning Unhealthy pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Startup probe failed: Get "http://10.132.0.160:15021/healthz/ready": dial tcp 10.132.0.160:15021: connect: connection refused 39m Normal AddedInterface pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Add eth0 [10.133.0.140/23] from ovn-kubernetes 39m Warning ProbeError pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Startup probe error: Get "http://10.132.0.160:15021/healthz/ready": dial tcp 10.132.0.160:15021: connect: connection refused... 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-api-b-c3486b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-api-b-c3486b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547 Error creating: pods "payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-2a8f18: serviceaccount "payload-processing-e2e-inf-b-2a8f18" not found 39m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547 Created pod: payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk 39m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-2a8f18-64646864cd Error creating: pods "payload-processing-e2e-inf-b-2a8f18-64646864cd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-2a8f18: serviceaccount "payload-processing-e2e-inf-b-2a8f18" not found 39m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-2a8f18 Scaled up replica set payload-processing-e2e-inf-b-2a8f18-64646864cd from 0 to 1 39m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-2a8f18-64646864cd Created pod: payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj 39m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-2a8f18 Scaled up replica set payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547 from 0 to 1 39m Normal AddedInterface pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Add eth0 [10.134.0.212/23] from ovn-kubernetes 39m Normal Started pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Container started 39m Normal Started pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Container started 39m Normal Created pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Container created 39m Normal Pulled pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Add eth0 [10.134.0.213/23] from ovn-kubernetes 39m Normal Pulled pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 39m Normal Created pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Container created 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-ccafe0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-ccafe0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 39m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-8afb26-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 39m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-8afb26-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Normal SuccessfulCreate replicaset/e2e-inf-b-e16297-openshift-default-7bd76cf86f Created pod: e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 38m Normal ScalingReplicaSet deployment/e2e-inf-b-e16297-openshift-default Scaled up replica set e2e-inf-b-e16297-openshift-default-7bd76cf86f from 0 to 1 38m Normal AddedLabel gateway/e2e-inf-b-e16297 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-e16297 38m Normal AddedInterface pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Add eth0 [10.134.0.214/23] from ovn-kubernetes 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-ccafe0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-ccafe0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-e16297-6dd54487fd Error creating: pods "payload-processing-e2e-inf-b-e16297-6dd54487fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e16297: serviceaccount "payload-processing-e2e-inf-b-e16297" not found 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-e16297 Scaled up replica set payload-pre-processing-e2e-inf-b-e16297-5b8559c79f from 0 to 1 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-e16297 Scaled up replica set payload-processing-e2e-inf-b-e16297-6dd54487fd from 0 to 1 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f Error creating: pods "payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e16297: serviceaccount "payload-processing-e2e-inf-b-e16297" not found 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-e16297-6dd54487fd Created pod: payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f Created pod: payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv 38m Normal Started pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Container started 38m Normal Started pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Container started 38m Normal Created pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Add eth0 [10.134.0.216/23] from ovn-kubernetes 38m Normal Pulled pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Pulled pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Created pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Container created 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Add eth0 [10.134.0.215/23] from ovn-kubernetes 38m Normal Created pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Container created 38m Normal Started pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Container started 38m Normal Pulled pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 38m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default Scaled down replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 4 to 3 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 3; reason: All metrics below target 38m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-tqm54 Stopping container istio-proxy 38m Warning ProbeError pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Startup probe error: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused... 38m Warning Unhealthy pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Startup probe failed: Get "http://10.134.0.190:15021/healthz/ready": dial tcp 10.134.0.190:15021: connect: connection refused 38m Normal Killing pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Stopping container istio-proxy 38m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt Stopping container istio-proxy 38m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 10 to 7 38m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-5c4lt 38m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 38m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 38m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx 38m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-m84l5 Stopping container istio-proxy 38m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-cdxvx Stopping container istio-proxy 38m Normal AddedLabel gateway/e2e-inf-a-802b8a Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-802b8a 38m Normal AddedInterface pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Add eth0 [10.132.0.174/23] from ovn-kubernetes 38m Normal ScalingReplicaSet deployment/e2e-inf-a-802b8a-openshift-default Scaled up replica set e2e-inf-a-802b8a-openshift-default-6f59fb88fc from 0 to 1 38m Warning ProbeError pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Readiness probe error: HTTP probe failed with statuscode: 503... 38m Normal SuccessfulCreate replicaset/e2e-inf-a-802b8a-openshift-default-6f59fb88fc Created pod: e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d 38m Warning Unhealthy pod/e2e-api-b-c3486b-openshift-default-7684fb8cf-zqnhk Readiness probe failed: HTTP probe failed with statuscode: 503 38m Normal ScalingReplicaSet deployment/e2e-inf-a-010013-openshift-default Scaled up replica set e2e-inf-a-010013-openshift-default-59977f9669 from 1 to 2 38m Normal SuccessfulCreate replicaset/e2e-inf-a-010013-openshift-default-59977f9669 Created pod: e2e-inf-a-010013-openshift-default-59977f9669-mx96q 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-010013-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 38m Normal Created pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Container created 38m Normal Started pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Container started 38m Normal Pulled pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Add eth0 [10.133.0.144/23] from ovn-kubernetes 38m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f Error creating: pods "payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-802b8a: serviceaccount "payload-processing-e2e-inf-a-802b8a" not found 38m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-802b8a Scaled up replica set payload-processing-e2e-inf-a-802b8a-5c4c6485b6 from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-802b8a-5c4c6485b6 Created pod: payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 38m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-802b8a-5c4c6485b6 Error creating: pods "payload-processing-e2e-inf-a-802b8a-5c4c6485b6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-802b8a: serviceaccount "payload-processing-e2e-inf-a-802b8a" not found 38m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-802b8a Scaled up replica set payload-pre-processing-e2e-inf-a-802b8a-845df8c77f from 0 to 1 38m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f Created pod: payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc 38m Normal Created pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Container created 38m Normal AddedInterface pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Add eth0 [10.134.0.218/23] from ovn-kubernetes 38m Normal Started pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Container started 38m Normal Pulled pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Add eth0 [10.134.0.217/23] from ovn-kubernetes 38m Normal Created pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Container created 38m Normal Pulled pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 38m Normal Started pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Container started 38m Warning Unhealthy pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Startup probe failed: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Startup probe error: Get "http://10.133.0.144:15021/healthz/ready": dial tcp 10.133.0.144:15021: connect: connection refused... 38m Normal SuccessfulCreate replicaset/e2e-inf-b-453511-openshift-default-9db66fcf5 Created pod: e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 38m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-453511-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 38m Normal ScalingReplicaSet deployment/e2e-inf-b-453511-openshift-default Scaled up replica set e2e-inf-b-453511-openshift-default-9db66fcf5 from 1 to 2 38m Normal Pulled pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 38m Normal AddedInterface pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Add eth0 [10.132.0.175/23] from ovn-kubernetes 38m Normal Created pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Container created 38m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-36fd3c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-36fd3c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 38m Normal Started pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Container started 38m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled up replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 0 to 1 38m Normal AddedLabel gateway/e2e-shared-b-w6-a914e9 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w6-a914e9 38m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s 38m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Add eth0 [10.134.0.219/23] from ovn-kubernetes 38m Warning Unhealthy pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Startup probe failed: Get "http://10.132.0.175:15021/healthz/ready": dial tcp 10.132.0.175:15021: connect: connection refused 38m Warning ProbeError pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Startup probe error: Get "http://10.132.0.175:15021/healthz/ready": dial tcp 10.132.0.175:15021: connect: connection refused... 37m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w6-a914e9 Scaled up replica set payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7 from 0 to 1 37m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7 Created pod: payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z 37m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7 Error creating: pods "payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9: serviceaccount "payload-processing-e2e-shared-b-w6-a914e9" not found 37m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w6-a914e9 Scaled up replica set payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd from 0 to 1 37m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd Created pod: payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm 37m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd Error creating: pods "payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w6-a914e9: serviceaccount "payload-processing-e2e-shared-b-w6-a914e9" not found 37m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Add eth0 [10.134.0.220/23] from ovn-kubernetes 37m Normal Pulled pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 37m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Add eth0 [10.134.0.221/23] from ovn-kubernetes 37m Normal Created pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Container created 37m Normal Started pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Container started 37m Normal Created pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Container created 37m Normal Started pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Container started 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-36fd3c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-36fd3c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Pulled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 Stopping container istio-proxy 37m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 37m Normal Started pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Container started 37m Normal Created pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Container created 37m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Startup probe error: Get "http://10.133.0.140:15021/healthz/ready": dial tcp 10.133.0.140:15021: connect: connection refused... 37m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Startup probe failed: Get "http://10.133.0.140:15021/healthz/ready": dial tcp 10.133.0.140:15021: connect: connection refused 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Normal Started pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Container started 37m Normal Pulled pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 37m Normal Created pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Container created 37m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-e16297-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-e16297-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 37m Warning Unhealthy pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Startup probe failed: Get "http://10.134.0.214:15021/healthz/ready": dial tcp 10.134.0.214:15021: connect: connection refused 37m Warning ProbeError pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Startup probe error: Get "http://10.134.0.214:15021/healthz/ready": dial tcp 10.134.0.214:15021: connect: connection refused... 36m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j 36m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Stopping container istio-proxy 36m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-xc56j Stopping container istio-proxy 36m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default Scaled down replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 4 to 3 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 3; reason: All metrics below target 36m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-36fd3c-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal ScalingReplicaSet deployment/e2e-inf-b-36fd3c-openshift-default Scaled up replica set e2e-inf-b-36fd3c-openshift-default-cc6c6c87f from 1 to 2 36m Normal SuccessfulCreate replicaset/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f Created pod: e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw 36m Normal AddedInterface pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Add eth0 [10.133.0.149/23] from ovn-kubernetes 36m Normal Created pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Container created 36m Normal Started pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Container started 36m Normal Pulled pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Readiness probe error: HTTP probe failed with statuscode: 503... 36m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-wlj2q Readiness probe failed: HTTP probe failed with statuscode: 503 36m Warning ProbeError pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Startup probe error: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Startup probe failed: Get "http://10.133.0.149:15021/healthz/ready": dial tcp 10.133.0.149:15021: connect: connection refused 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-397179-openshift-default New size: 2; reason: All metrics below target 36m Normal ScalingReplicaSet deployment/e2e-worker-w2-397179-openshift-default Scaled down replica set e2e-worker-w2-397179-openshift-default-bc795b5d9 from 3 to 2 36m Normal SuccessfulDelete replicaset/e2e-worker-w2-397179-openshift-default-bc795b5d9 Deleted pod: e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx 36m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmnkx Stopping container istio-proxy 36m Normal AddedLabel gateway/e2e-inf-b-68b893 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-68b893 36m Normal AddedInterface pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Add eth0 [10.134.0.225/23] from ovn-kubernetes 36m Normal ScalingReplicaSet deployment/e2e-inf-b-68b893-openshift-default Scaled up replica set e2e-inf-b-68b893-openshift-default-5d65c9cc49 from 0 to 1 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-e16297-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-e16297-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 36m Normal SuccessfulCreate replicaset/e2e-inf-b-68b893-openshift-default-5d65c9cc49 Created pod: e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 36m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-68b893 Scaled up replica set payload-processing-e2e-inf-b-68b893-6ffc449d94 from 0 to 1 36m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-68b893-6ffc449d94 Error creating: pods "payload-processing-e2e-inf-b-68b893-6ffc449d94-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-68b893: serviceaccount "payload-processing-e2e-inf-b-68b893" not found 36m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-68b893 Scaled up replica set payload-pre-processing-e2e-inf-b-68b893-db65656ff from 0 to 1 36m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-68b893-db65656ff Created pod: payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb 36m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-68b893-6ffc449d94 Created pod: payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 36m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-68b893-db65656ff Error creating: pods "payload-pre-processing-e2e-inf-b-68b893-db65656ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-68b893: serviceaccount "payload-processing-e2e-inf-b-68b893" not found 36m Normal AddedInterface pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Add eth0 [10.134.0.227/23] from ovn-kubernetes 36m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Add eth0 [10.134.0.226/23] from ovn-kubernetes 36m Normal Started pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Container started 36m Normal Started pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Container started 36m Normal Created pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Container created 36m Normal Pulled pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Created pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Container created 36m Normal Pulled pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 36m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Add eth0 [10.133.0.150/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 36m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Container created 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 36m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Add eth0 [10.134.0.229/23] from ovn-kubernetes 36m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td 36m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js 36m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Add eth0 [10.132.0.178/23] from ovn-kubernetes 36m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled up replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 1 to 4 36m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Container started 36m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Container created 36m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Container started 36m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Container started 36m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Container created 36m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Readiness probe failed: Get "http://10.134.0.219:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Readiness probe error: Get "http://10.134.0.219:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 36m Normal Created pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Container created 36m Normal Pulled pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Started pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Container started 36m Normal Killing pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Stopping container payload-processing 36m Normal Killing pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Stopping container payload-pre-processing 36m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-356cae-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-356cae-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 36m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Startup probe failed: Get "http://10.132.0.178:15021/healthz/ready": dial tcp 10.132.0.178:15021: connect: connection refused 36m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Startup probe failed: Get "http://10.133.0.150:15021/healthz/ready": dial tcp 10.133.0.150:15021: connect: connection refused 36m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Container created 36m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Startup probe error: Get "http://10.132.0.178:15021/healthz/ready": dial tcp 10.132.0.178:15021: connect: connection refused... 36m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Container started 36m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Startup probe error: Get "http://10.134.0.229:15021/healthz/ready": dial tcp 10.134.0.229:15021: connect: connection refused... 36m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Startup probe error: Get "http://10.133.0.150:15021/healthz/ready": dial tcp 10.133.0.150:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Startup probe failed: Get "http://10.134.0.229:15021/healthz/ready": dial tcp 10.134.0.229:15021: connect: connection refused 36m Warning Unhealthy pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Startup probe failed: Get "http://10.133.0.127:15021/healthz/ready": dial tcp 10.133.0.127:15021: connect: connection refused 36m Warning ProbeError pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Startup probe error: Get "http://10.133.0.127:15021/healthz/ready": dial tcp 10.133.0.127:15021: connect: connection refused... 36m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Startup probe failed: Get "http://10.134.0.219:15021/healthz/ready": dial tcp 10.134.0.219:15021: connect: connection refused 36m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Startup probe error: Get "http://10.134.0.219:15021/healthz/ready": dial tcp 10.134.0.219:15021: connect: connection refused... 36m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-8afb26-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 36m Normal SuccessfulCreate replicaset/e2e-inf-b-8afb26-openshift-default-6d8b784f7f Created pod: e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 36m Normal ScalingReplicaSet deployment/e2e-inf-b-8afb26-openshift-default Scaled up replica set e2e-inf-b-8afb26-openshift-default-6d8b784f7f from 1 to 2 36m Normal Killing pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-lkbnq Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-5fhzz Stopping container istio-proxy 36m Normal Killing pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-t75f7 Stopping container istio-proxy 36m Normal Pulled pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 36m Normal Created pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Container created 36m Normal Started pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Container started 36m Normal AddedInterface pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Add eth0 [10.132.0.179/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Add eth0 [10.134.0.232/23] from ovn-kubernetes 35m Normal ScalingReplicaSet deployment/e2e-inf-b-2a8f18-openshift-default Scaled up replica set e2e-inf-b-2a8f18-openshift-default-74685fbccc from 1 to 2 35m Normal SuccessfulCreate replicaset/e2e-inf-b-2a8f18-openshift-default-74685fbccc Created pod: e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Normal Started pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Container started 35m Normal Created pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Container created 35m Warning Unhealthy pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Startup probe failed: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused 35m Warning ProbeError pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Startup probe error: Get "http://10.132.0.179:15021/healthz/ready": dial tcp 10.132.0.179:15021: connect: connection refused... 35m Normal ScalingReplicaSet deployment/e2e-inf-b-e16297-openshift-default Scaled up replica set e2e-inf-b-e16297-openshift-default-7bd76cf86f from 1 to 2 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-e16297-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 35m Normal SuccessfulCreate replicaset/e2e-inf-b-e16297-openshift-default-7bd76cf86f Created pod: e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l 35m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Startup probe failed: Get "http://10.134.0.232:15021/healthz/ready": dial tcp 10.134.0.232:15021: connect: connection refused 35m Normal AddedInterface pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Add eth0 [10.132.0.180/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-1f8eb4-767f87bd59-4xfch Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Normal Started pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Container started 35m Normal Created pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Container created 35m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Startup probe error: Get "http://10.134.0.232:15021/healthz/ready": dial tcp 10.134.0.232:15021: connect: connection refused... 35m Normal Killing pod/payload-processing-e2e-inf-b-36fd3c-7895ccc8dd-w4jwf Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Stopping container payload-pre-processing 35m Warning Unhealthy pod/payload-processing-e2e-inf-a-1f8eb4-654c4687f6-b6tfq Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning Unhealthy pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Startup probe failed: Get "http://10.132.0.180:15021/healthz/ready": dial tcp 10.132.0.180:15021: connect: connection refused 35m Warning ProbeError pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Startup probe error: Get "http://10.132.0.180:15021/healthz/ready": dial tcp 10.132.0.180:15021: connect: connection refused... 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-356cae-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-356cae-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 35m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Container started 35m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Add eth0 [10.133.0.151/23] from ovn-kubernetes 35m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Add eth0 [10.132.0.183/23] from ovn-kubernetes 35m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww 35m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Container created 35m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr 35m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled up replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 4 to 6 35m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-36fd3c-86dc48cdc5-q8nbc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 35m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Container started 35m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Container created 35m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Readiness probe failed: Get "http://10.134.0.115:8080/ready": read tcp 10.134.0.2:36872->10.134.0.115:8080: read: connection reset by peer 35m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-nxlpl 35m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Readiness probe error: Get "http://10.134.0.115:8080/ready": read tcp 10.134.0.2:36872->10.134.0.115:8080: read: connection reset by peer... 35m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-nxlpl Stopping container discovery 35m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Startup probe error: Get "http://10.132.0.183:15021/healthz/ready": dial tcp 10.132.0.183:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Startup probe failed: Get "http://10.132.0.183:15021/healthz/ready": dial tcp 10.132.0.183:15021: connect: connection refused 35m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Startup probe error: Get "http://10.133.0.151:15021/healthz/ready": dial tcp 10.133.0.151:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Startup probe failed: Get "http://10.133.0.151:15021/healthz/ready": dial tcp 10.133.0.151:15021: connect: connection refused 35m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-tbcpj 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 3; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default Scaled down replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 4 to 3 35m Normal Pulled pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 35m Normal Started pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Container started 35m Normal Created pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Container created 35m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-2f6fed-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-2f6fed-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-d55037-openshift-default New size: 2; reason: All metrics below target 35m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Stopping container istio-proxy 35m Normal SuccessfulDelete replicaset/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 Deleted pod: e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-cff4ac-openshift-default New size: 2; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/e2e-worker-w4-d55037-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w4-d55037-openshift-default-77bc79dc85 from 3 to 2 35m Normal ScalingReplicaSet deployment/e2e-worker-w1-cff4ac-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w1-cff4ac-openshift-default-75448d8ddf from 3 to 2 35m Normal SuccessfulDelete replicaset/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf Deleted pod: e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf 35m Normal SuccessfulDelete replicaset/e2e-worker-w4-d55037-openshift-default-77bc79dc85 Deleted pod: e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 35m Normal ScalingReplicaSet deployment/e2e-worker-w3-889ca9-openshift-default Scaled down replica set e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5 from 3 to 2 35m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-tg8hk Stopping container istio-proxy 35m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-889ca9-openshift-default New size: 2; reason: All metrics below target 35m Warning ProbeError pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Startup probe error: Get "http://10.132.0.168:15021/healthz/ready": dial tcp 10.132.0.168:15021: connect: connection refused... 35m Warning Unhealthy pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Startup probe failed: Get "http://10.132.0.168:15021/healthz/ready": dial tcp 10.132.0.168:15021: connect: connection refused 35m Normal Killing pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Stopping container payload-processing 35m Normal Killing pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Stopping container payload-pre-processing 35m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Readiness probe error: HTTP probe failed with statuscode: 503... 35m Warning Unhealthy pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 35m Normal Killing pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-2t8pn Stopping container istio-proxy 35m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: All metrics below target 35m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 7 to 6 35m Normal Killing pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Stopping container istio-proxy 35m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Stopping container istio-proxy 35m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-34722e-openshift-default Scaled up replica set e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 from 0 to 1 35m Normal AddedLabel gateway/e2e-shared-a-w5-34722e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w5-34722e 35m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 Created pod: e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v 35m Normal AddedInterface pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Add eth0 [10.134.0.237/23] from ovn-kubernetes 35m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Readiness probe failed: HTTP probe failed with statuscode: 503 35m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 Readiness probe error: HTTP probe failed with statuscode: 503... 34m Warning Unhealthy pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Readiness probe failed: Get "http://10.133.0.149:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw Readiness probe error: Get "http://10.133.0.149:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7 Created pod: payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz 34m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w5-34722e-54578dd88 Created pod: payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q 34m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7 Error creating: pods "payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-34722e: serviceaccount "payload-processing-e2e-shared-a-w5-34722e" not found 34m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w5-34722e Scaled up replica set payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7 from 0 to 1 34m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w5-34722e-54578dd88 Error creating: pods "payload-processing-e2e-shared-a-w5-34722e-54578dd88-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w5-34722e: serviceaccount "payload-processing-e2e-shared-a-w5-34722e" not found 34m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w5-34722e Scaled up replica set payload-processing-e2e-shared-a-w5-34722e-54578dd88 from 0 to 1 34m Normal Pulled pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Created pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Container created 34m Normal Started pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Container started 34m Normal Created pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Container created 34m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Add eth0 [10.134.0.239/23] from ovn-kubernetes 34m Normal AddedInterface pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Add eth0 [10.134.0.238/23] from ovn-kubernetes 34m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 34m Normal Started pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Container started 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-010013-744897b695-fqc2t Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning Unhealthy pod/payload-processing-e2e-inf-a-010013-65dc787d94-c5lvn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w6-2f6fed-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w6-2f6fed-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 34m Normal Killing pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Stopping container payload-processing 34m Normal Killing pod/e2e-inf-a-010013-openshift-default-59977f9669-scchv Stopping container istio-proxy 34m Normal Killing pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Stopping container istio-proxy 34m Warning ProbeError pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Readiness probe error: HTTP probe failed with statuscode: 503... 34m Warning Unhealthy pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q Readiness probe failed: HTTP probe failed with statuscode: 503 34m Normal Killing pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Stopping container payload-pre-processing 34m Normal Killing pod/payload-processing-e2e-inf-b-8afb26-864b7b9f48-g7zpn Stopping container payload-processing 34m Warning Unhealthy pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Readiness probe failed: Get "http://10.134.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning ProbeError pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Readiness probe error: Get "http://10.134.0.225:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 34m Normal SuccessfulCreate replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Created pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 34m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled up replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 6 to 7 34m Normal AddedInterface pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Add eth0 [10.134.0.245/23] from ovn-kubernetes 34m Normal Pulled pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Pulled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Started pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Container started 34m Normal Created pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Container created 34m Normal Created pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Container created 34m Normal Started pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Container started 34m Normal ScalingReplicaSet deployment/e2e-inf-a-356cae-openshift-default Scaled up replica set e2e-inf-a-356cae-openshift-default-5bd47d6d5 from 1 to 2 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-a-356cae-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 34m Normal SuccessfulCreate replicaset/e2e-inf-a-356cae-openshift-default-5bd47d6d5 Created pod: e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 34m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-c7f962-85df567dd7-mx6zs Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Normal AddedInterface pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Add eth0 [10.134.0.246/23] from ovn-kubernetes 34m Normal Pulled pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Normal Created pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Container created 34m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-2a8f18-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 34m Normal SuccessfulCreate replicaset/e2e-inf-b-2a8f18-openshift-default-74685fbccc Created pod: e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw 34m Normal ScalingReplicaSet deployment/e2e-inf-b-2a8f18-openshift-default Scaled up replica set e2e-inf-b-2a8f18-openshift-default-74685fbccc from 2 to 3 34m Normal Started pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Container started 34m Warning ProbeError pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Startup probe error: Get "http://10.134.0.225:15021/healthz/ready": dial tcp 10.134.0.225:15021: connect: connection refused... 34m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Startup probe error: Get "http://10.134.0.245:15021/healthz/ready": dial tcp 10.134.0.245:15021: connect: connection refused... 34m Normal Pulled pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 34m Warning Unhealthy pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Startup probe failed: Get "http://10.134.0.225:15021/healthz/ready": dial tcp 10.134.0.225:15021: connect: connection refused 34m Normal AddedInterface pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Add eth0 [10.132.0.184/23] from ovn-kubernetes 34m Warning Unhealthy pod/payload-processing-e2e-inf-a-c7f962-6886679566-84khk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 34m Normal Started pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Container started 34m Normal Created pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Container created 34m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Startup probe failed: Get "http://10.134.0.245:15021/healthz/ready": dial tcp 10.134.0.245:15021: connect: connection refused 34m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 34m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Startup probe failed: Get "http://10.132.0.184:15021/healthz/ready": dial tcp 10.132.0.184:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Startup probe error: Get "http://10.132.0.184:15021/healthz/ready": dial tcp 10.132.0.184:15021: connect: connection refused... 34m Warning Unhealthy pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Startup probe failed: Get "http://10.134.0.246:15021/healthz/ready": dial tcp 10.134.0.246:15021: connect: connection refused 34m Warning ProbeError pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Startup probe error: Get "http://10.134.0.246:15021/healthz/ready": dial tcp 10.134.0.246:15021: connect: connection refused... 34m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw 34m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-2n5mw Stopping container istio-proxy 34m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 6 to 5 34m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-8afb26-5b6cb6d879-w4dhn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal Killing pod/e2e-inf-a-c7f962-openshift-default-858f5c9bf-zhtcs Stopping container istio-proxy 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Killing pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-6f764 Stopping container istio-proxy 33m Normal Killing pod/e2e-inf-b-8afb26-openshift-default-6d8b784f7f-m9tn5 Stopping container istio-proxy 33m Normal Killing pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Stopping container payload-pre-processing 33m Normal Killing pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Stopping container payload-processing 33m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf Created pod: e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll 33m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-262c10-openshift-default Scaled up replica set e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf from 0 to 1 33m Normal AddedLabel gateway/e2e-shared-a-w4-262c10 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w4-262c10 33m Normal AddedInterface pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Add eth0 [10.134.0.250/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w4-262c10-bccc5986d Created pod: payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w4-262c10-bccc5986d Error creating: pods "payload-processing-e2e-shared-a-w4-262c10-bccc5986d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-262c10: serviceaccount "payload-processing-e2e-shared-a-w4-262c10" not found 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc Error creating: pods "payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w4-262c10: serviceaccount "payload-processing-e2e-shared-a-w4-262c10" not found 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w4-262c10 Scaled up replica set payload-processing-e2e-shared-a-w4-262c10-bccc5986d from 0 to 1 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc Created pod: payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w4-262c10 Scaled up replica set payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc from 0 to 1 33m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Add eth0 [10.134.0.251/23] from ovn-kubernetes 33m Normal Pulled pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Add eth0 [10.134.0.252/23] from ovn-kubernetes 33m Normal Created pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Container created 33m Normal Started pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Container started 33m Normal Created pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Container created 33m Normal Started pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Container started 33m Normal SuccessfulDelete replicaset/e2e-worker-w6-83e83e-openshift-default-796ccd4c65 Deleted pod: e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 33m Normal ScalingReplicaSet deployment/e2e-worker-w6-83e83e-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w6-83e83e-openshift-default-796ccd4c65 from 3 to 2 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-83e83e-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 33m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Stopping container istio-proxy 33m Normal Started pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Container started 33m Normal Pulled pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal Created pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Container created 33m Normal Created pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Container created 33m Normal Started pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Container started 33m Normal Pulled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Warning ProbeError pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-kzkp5 Readiness probe failed: HTTP probe failed with statuscode: 503 33m Normal AddedLabel gateway/e2e-shared-b-w5-0dba31 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w5-0dba31 33m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0dba31-openshift-default Scaled up replica set e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc from 0 to 1 33m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc Created pod: e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Stopping container istio-proxy 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-453511-6778fc7699-vc29t Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Normal AddedInterface pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Add eth0 [10.134.0.254/23] from ovn-kubernetes 33m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 33m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-672ql 33m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 5 to 4 33m Warning Unhealthy pod/payload-processing-e2e-inf-b-453511-5d55648d-c4bx2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 33m Warning Unhealthy pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Startup probe failed: Get "http://10.132.0.174:15021/healthz/ready": dial tcp 10.132.0.174:15021: connect: connection refused 33m Warning ProbeError pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Startup probe error: Get "http://10.132.0.174:15021/healthz/ready": dial tcp 10.132.0.174:15021: connect: connection refused... 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Readiness probe error: HTTP probe failed with statuscode: 503... 33m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-672ql Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Startup probe error: Get "http://10.134.0.237:15021/healthz/ready": dial tcp 10.134.0.237:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Startup probe failed: Get "http://10.134.0.237:15021/healthz/ready": dial tcp 10.134.0.237:15021: connect: connection refused 33m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w5-0dba31 Scaled up replica set payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd from 0 to 1 33m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w5-0dba31 Scaled up replica set payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747 from 0 to 1 33m Normal Created pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Container created 33m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd Error creating: pods "payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31: serviceaccount "payload-processing-e2e-shared-b-w5-0dba31" not found 33m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd Created pod: payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv 33m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Add eth0 [10.132.0.185/23] from ovn-kubernetes 33m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747 Created pod: payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs 33m Normal Started pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Container started 33m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747 Error creating: pods "payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w5-0dba31: serviceaccount "payload-processing-e2e-shared-b-w5-0dba31" not found 33m Normal AddedInterface pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Add eth0 [10.134.0.255/23] from ovn-kubernetes 33m Normal Created pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Container created 33m Normal Started pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Container started 33m Normal Pulled pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 33m Normal Killing pod/payload-processing-e2e-inf-a-ccafe0-7bbc9b8996-gmnr5 Stopping container payload-processing 33m Normal Killing pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Stopping container payload-pre-processing 33m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 33m Normal SuccessfulCreate replicaset/e2e-inf-b-68b893-openshift-default-5d65c9cc49 Created pod: e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt 33m Normal ScalingReplicaSet deployment/e2e-inf-b-68b893-openshift-default Scaled up replica set e2e-inf-b-68b893-openshift-default-5d65c9cc49 from 1 to 2 33m Normal Pulled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 33m Normal AddedInterface pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Add eth0 [10.132.0.186/23] from ovn-kubernetes 33m Normal Started pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Container started 33m Normal Created pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Container created 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 33m Warning ProbeError pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Startup probe error: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused... 33m Warning Unhealthy pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Startup probe failed: Get "http://10.132.0.186:15021/healthz/ready": dial tcp 10.132.0.186:15021: connect: connection refused 32m Normal Killing pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s8nz6 Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-b-453511-openshift-default-9db66fcf5-s69jk Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-a-ccafe0-openshift-default-6f9c49b9f9-s6h8f Stopping container istio-proxy 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-ccafe0-5988c5c447-np4pn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-inf-a-802b8a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-inf-a-802b8a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Normal Killing pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Stopping container payload-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Stopping container payload-pre-processing 32m Normal Killing pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Stopping container payload-pre-processing 32m Normal Killing pod/payload-processing-e2e-inf-a-356cae-795777684b-ftbfw Stopping container payload-processing 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-2a8f18-7b564fd547-dlsdk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-processing-e2e-inf-b-2a8f18-64646864cd-ttltj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-356cae-7765997747-nfphg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 32m Normal Killing pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Stopping container istio-proxy 32m Normal Killing pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Stopping container istio-proxy 32m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-2kbxj Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p Readiness probe error: HTTP probe failed with statuscode: 503... 32m Warning Unhealthy pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Readiness probe failed: HTTP probe failed with statuscode: 503 32m Warning ProbeError pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-dn5qw Readiness probe error: HTTP probe failed with statuscode: 503... 32m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-674082-openshift-default Scaled up replica set e2e-shared-b-w4-674082-openshift-default-577d994559 from 0 to 1 32m Normal AddedLabel gateway/e2e-shared-b-w4-674082 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w4-674082 32m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-674082-openshift-default-577d994559 Created pod: e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr 32m Normal AddedInterface pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Add eth0 [10.134.1.6/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 32m Normal AddedInterface pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Add eth0 [10.132.0.191/23] from ovn-kubernetes 32m Normal AddedLabel gateway/e2e-ipp-a-6b3855 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-6b3855 32m Normal ScalingReplicaSet deployment/e2e-ipp-a-6b3855-openshift-default Scaled up replica set e2e-ipp-a-6b3855-openshift-default-5f877686d5 from 0 to 1 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w4-674082 Scaled up replica set payload-processing-e2e-shared-b-w4-674082-6c4df5884d from 0 to 1 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w4-674082 Scaled up replica set payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59 from 0 to 1 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59 Error creating: pods "payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-674082: serviceaccount "payload-processing-e2e-shared-b-w4-674082" not found 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w4-674082-6c4df5884d Created pod: payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt 31m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w4-674082-6c4df5884d Error creating: pods "payload-processing-e2e-shared-b-w4-674082-6c4df5884d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w4-674082: serviceaccount "payload-processing-e2e-shared-b-w4-674082" not found 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59 Created pod: payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 31m Normal Started pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Container started 31m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-6b3855 Scaled up replica set payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f from 0 to 1 31m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Add eth0 [10.132.0.192/23] from ovn-kubernetes 31m Normal Killing pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-n2mv9 Stopping container istio-proxy 31m Normal Killing pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Stopping container istio-proxy 31m Normal AddedInterface pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Add eth0 [10.132.0.193/23] from ovn-kubernetes 31m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-6b3855 Scaled up replica set payload-processing-e2e-ipp-a-6b3855-c44b886c4 from 0 to 1 31m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-6b3855-c44b886c4 Created pod: payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr 31m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-6b3855-c44b886c4 Error creating: pods "payload-processing-e2e-ipp-a-6b3855-c44b886c4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-6b3855: serviceaccount "payload-processing-e2e-ipp-a-6b3855" not found 31m Normal Pulled pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Created pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Container created 31m Normal Created pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Container created 31m Normal Started pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Container started 31m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f Created pod: payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz 31m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f Error creating: pods "payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-6b3855: serviceaccount "payload-processing-e2e-ipp-a-6b3855" not found 31m Normal Created pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Container created 31m Normal AddedInterface pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Add eth0 [10.132.0.194/23] from ovn-kubernetes 31m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Add eth0 [10.134.1.7/23] from ovn-kubernetes 31m Normal Pulled pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 31m Normal Started pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Container started 31m Warning ProbeError pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Readiness probe error: HTTP probe failed with statuscode: 503... 31m Normal Created pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Container created 31m Warning Unhealthy pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w Readiness probe failed: HTTP probe failed with statuscode: 503 31m Normal Started pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Container started 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Normal Killing pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Stopping container payload-pre-processing 31m Normal Killing pod/payload-processing-e2e-inf-b-e16297-6dd54487fd-vnx5q Stopping container payload-processing 31m Normal Created pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Container created 31m Normal Pulled pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Started pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Container started 31m Normal Started pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Container started 31m Normal Created pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Container created 31m Normal Pulled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Readiness probe failed: Get "http://10.134.0.254:15021/healthz/ready": read tcp 10.134.0.2:39490->10.134.0.254:15021: read: connection reset by peer 31m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Readiness probe error: Get "http://10.134.0.254:15021/healthz/ready": read tcp 10.134.0.2:39490->10.134.0.254:15021: read: connection reset by peer... 31m Normal Pulled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Container created 31m Normal Started pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Container started 31m Warning ProbeError pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Startup probe error: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Startup probe failed: Get "http://10.132.0.106:15021/healthz/ready": dial tcp 10.132.0.106:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Startup probe error: Get "http://10.134.0.250:15021/healthz/ready": dial tcp 10.134.0.250:15021: connect: connection refused... 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 31m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f 31m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled up replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 1 to 2 31m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Startup probe failed: Get "http://10.134.0.250:15021/healthz/ready": dial tcp 10.134.0.250:15021: connect: connection refused 31m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Container started 31m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Container created 31m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Add eth0 [10.132.0.195/23] from ovn-kubernetes 31m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Startup probe failed: Get "http://10.134.0.254:15021/healthz/ready": dial tcp 10.134.0.254:15021: connect: connection refused 31m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Startup probe error: Get "http://10.134.0.254:15021/healthz/ready": dial tcp 10.134.0.254:15021: connect: connection refused... 31m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-e16297-5b8559c79f-dn4wv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 31m Normal ScalingReplicaSet deployment/e2e-inf-b-68b893-openshift-default Scaled up replica set e2e-inf-b-68b893-openshift-default-5d65c9cc49 from 2 to 3 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-inf-b-68b893-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 31m Normal SuccessfulCreate replicaset/e2e-inf-b-68b893-openshift-default-5d65c9cc49 Created pod: e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 31m Normal AddedInterface pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Add eth0 [10.133.0.153/23] from ovn-kubernetes 31m Normal Started pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Container started 31m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Startup probe error: Get "http://10.132.0.195:15021/healthz/ready": dial tcp 10.132.0.195:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Startup probe failed: Get "http://10.132.0.195:15021/healthz/ready": dial tcp 10.132.0.195:15021: connect: connection refused 31m Normal Pulled pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 31m Normal Created pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Container created 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning ProbeError pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Startup probe error: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused... 31m Warning Unhealthy pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Startup probe failed: Get "http://10.133.0.153:15021/healthz/ready": dial tcp 10.133.0.153:15021: connect: connection refused 31m Normal SuccessfulDelete replicaset/e2e-worker-w5-0b21ba-openshift-default-7776bf6747 Deleted pod: e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 31m Normal SuccessfulDelete replicaset/e2e-worker-w0-b21507-openshift-default-7dcf4797cf Deleted pod: e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 31m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Stopping container istio-proxy 31m Normal ScalingReplicaSet deployment/e2e-worker-w0-b21507-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w0-b21507-openshift-default-7dcf4797cf from 3 to 2 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-0b21ba-openshift-default New size: 2; reason: All metrics below target 31m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-b21507-openshift-default (combined from similar events): New size: 2; reason: All metrics below target 31m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Stopping container istio-proxy 31m Normal ScalingReplicaSet deployment/e2e-worker-w5-0b21ba-openshift-default (combined from similar events): Scaled down replica set e2e-worker-w5-0b21ba-openshift-default-7776bf6747 from 3 to 2 31m Warning ProbeError pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 Readiness probe failed: HTTP probe failed with statuscode: 503 31m Warning ProbeError pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Readiness probe error: HTTP probe failed with statuscode: 503... 31m Warning Unhealthy pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-2m7f8 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w2-44bc69-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w2-44bc69-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal Killing pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-jnvm5 Stopping container istio-proxy 30m Normal Killing pod/e2e-inf-b-e16297-openshift-default-7bd76cf86f-n8h7l Stopping container istio-proxy 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 30m Normal Killing pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Stopping container istio-proxy 30m Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Deleted pod: maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 30m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 4 to 3 30m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-34722e-openshift-default Scaled up replica set e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 from 1 to 2 30m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 Created pod: e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 30m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 30m Normal Created pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Container created 30m Normal Started pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Container started 30m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph 30m Normal Pulled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm 30m Normal ScalingReplicaSet deployment/e2e-ipp-a-6b3855-openshift-default Scaled up replica set e2e-ipp-a-6b3855-openshift-default-5f877686d5 from 1 to 4 30m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l 30m Normal AddedInterface pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Add eth0 [10.132.0.198/23] from ovn-kubernetes 30m Normal Created pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Container created 30m Normal Pulled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Add eth0 [10.134.1.13/23] from ovn-kubernetes 30m Normal Started pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Container started 30m Normal Started pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Container started 30m Normal Created pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Container created 30m Normal Created pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Container created 30m Normal AddedInterface pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Add eth0 [10.134.1.14/23] from ovn-kubernetes 30m Normal Pulled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Started pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Container started 30m Normal Pulled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal AddedInterface pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Add eth0 [10.133.0.154/23] from ovn-kubernetes 30m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Readiness probe failed: HTTP probe failed with statuscode: 503 30m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Readiness probe failed: Get "http://10.134.1.6:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 30m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Readiness probe error: Get "http://10.132.0.191:15021/healthz/ready": read tcp 10.132.0.2:44188->10.132.0.191:15021: read: connection reset by peer... 30m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Readiness probe error: Get "http://10.134.1.6:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 30m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 Readiness probe error: HTTP probe failed with statuscode: 503... 30m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Readiness probe failed: Get "http://10.132.0.191:15021/healthz/ready": read tcp 10.132.0.2:44188->10.132.0.191:15021: read: connection reset by peer 30m Normal Created pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Container created 30m Normal Pulled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Started pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Container started 30m Normal Pulled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 30m Normal Created pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Container created 30m Normal Started pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Container started 30m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Startup probe failed: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused 30m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Startup probe error: Get "http://10.133.0.154:15021/healthz/ready": dial tcp 10.133.0.154:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Startup probe failed: Get "http://10.134.1.13:15021/healthz/ready": dial tcp 10.134.1.13:15021: connect: connection refused 30m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Startup probe error: Get "http://10.134.1.13:15021/healthz/ready": dial tcp 10.134.1.13:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Startup probe failed: Get "http://10.132.0.198:15021/healthz/ready": dial tcp 10.132.0.198:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Startup probe error: Get "http://10.132.0.198:15021/healthz/ready": dial tcp 10.132.0.198:15021: connect: connection refused... 30m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Startup probe error: Get "http://10.134.1.14:15021/healthz/ready": dial tcp 10.134.1.14:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Startup probe failed: Get "http://10.134.1.14:15021/healthz/ready": dial tcp 10.134.1.14:15021: connect: connection refused 30m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Startup probe failed: Get "http://10.132.0.191:15021/healthz/ready": dial tcp 10.132.0.191:15021: connect: connection refused 30m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Startup probe error: Get "http://10.132.0.191:15021/healthz/ready": dial tcp 10.132.0.191:15021: connect: connection refused... 30m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Startup probe failed: Get "http://10.134.1.6:15021/healthz/ready": dial tcp 10.134.1.6:15021: connect: connection refused 30m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Startup probe error: Get "http://10.134.1.6:15021/healthz/ready": dial tcp 10.134.1.6:15021: connect: connection refused... 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Normal Killing pod/payload-pre-processing-e2e-ipp-a-6b3855-7b84b8579f-m67wz Stopping container payload-pre-processing 30m Normal Killing pod/payload-processing-e2e-ipp-a-6b3855-c44b886c4-rbvqr Stopping container payload-processing 30m Normal Killing pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Stopping container payload-processing 30m Normal Killing pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Stopping container payload-pre-processing 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 30m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 30m Warning Unhealthy pod/payload-processing-e2e-inf-a-802b8a-5c4c6485b6-dwdk7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 30m Warning Unhealthy pod/payload-pre-processing-e2e-inf-a-802b8a-845df8c77f-svkjc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal AddedLabel gateway/e2e-ipp-a-60a531 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-60a531 29m Normal SuccessfulCreate replicaset/e2e-ipp-a-60a531-openshift-default-6c44b77d55 Created pod: e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 29m Normal ScalingReplicaSet deployment/e2e-ipp-a-60a531-openshift-default Scaled up replica set e2e-ipp-a-60a531-openshift-default-6c44b77d55 from 0 to 1 29m Normal Killing pod/e2e-inf-a-802b8a-openshift-default-6f59fb88fc-c9p5d Stopping container istio-proxy 29m Normal AddedInterface pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Add eth0 [10.134.1.20/23] from ovn-kubernetes 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd Error creating: pods "payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-6b3855: serviceaccount "payload-processing-e2e-ipp-a-6b3855" not found 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-6b3855 Scaled up replica set payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87 from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd Created pod: payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87 Error creating: pods "payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-6b3855: serviceaccount "payload-processing-e2e-ipp-a-6b3855" not found 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87 Created pod: payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-6b3855 Scaled up replica set payload-processing-e2e-ipp-a-6b3855-555c7f6cbd from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-60a531 Scaled up replica set payload-processing-e2e-ipp-a-60a531-764f46b8bd from 0 to 1 29m Normal Pulling pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 29m Normal AddedInterface pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Add eth0 [10.134.1.23/23] from ovn-kubernetes 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Add eth0 [10.132.0.199/23] from ovn-kubernetes 29m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-60a531-764f46b8bd Created pod: payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-60a531-764f46b8bd Error creating: pods "payload-processing-e2e-ipp-a-60a531-764f46b8bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-60a531: serviceaccount "payload-processing-e2e-ipp-a-60a531" not found 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-60a531 Scaled up replica set payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c Created pod: payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c Error creating: pods "payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-60a531: serviceaccount "payload-processing-e2e-ipp-a-60a531" not found 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.153s (2.153s including waiting). Image size: 166612641 bytes. 29m Normal Started pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Container started 29m Normal Created pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Container created 29m Normal Started pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Container started 29m Normal Created pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Container created 29m Normal AddedInterface pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Add eth0 [10.132.0.201/23] from ovn-kubernetes 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Pulled pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.142s (2.142s including waiting). Image size: 166612641 bytes. 29m Normal Created pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Container created 29m Normal Started pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Container started 29m Normal Pulled pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Container started 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Add eth0 [10.132.0.200/23] from ovn-kubernetes 29m Normal Created pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Container created 29m Normal Killing pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Stopping container payload-pre-processing 29m Normal Killing pod/payload-processing-e2e-inf-b-68b893-6ffc449d94-dgnq6 Stopping container payload-processing 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled up replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 2 to 3 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 29m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Container started 29m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Container created 29m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Add eth0 [10.133.0.155/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w3-e15c99-openshift-default Scaled up replica set e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7 from 1 to 2 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w3-e15c99-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7 Created pod: e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h 29m Normal AddedInterface pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Add eth0 [10.133.0.156/23] from ovn-kubernetes 29m Normal Pulled pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Container created 29m Normal Started pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Container started 29m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Startup probe failed: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Startup probe error: Get "http://10.133.0.155:15021/healthz/ready": dial tcp 10.133.0.155:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Startup probe failed: Get "http://10.133.0.156:15021/healthz/ready": dial tcp 10.133.0.156:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Startup probe error: Get "http://10.133.0.156:15021/healthz/ready": dial tcp 10.133.0.156:15021: connect: connection refused... 29m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm 29m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled up replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 0 to 1 29m Normal AddedLabel gateway/e2e-ipp-b-45607d Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-45607d 29m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Add eth0 [10.134.1.26/23] from ovn-kubernetes 29m Warning Unhealthy pod/payload-pre-processing-e2e-inf-b-68b893-db65656ff-2d6nb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 29m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9 Created pod: payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m 29m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-45607d Scaled up replica set payload-processing-e2e-ipp-b-45607d-c7bcdf84b from 0 to 1 29m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-45607d-c7bcdf84b Created pod: payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 29m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-45607d-c7bcdf84b Error creating: pods "payload-processing-e2e-ipp-b-45607d-c7bcdf84b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-45607d: serviceaccount "payload-processing-e2e-ipp-b-45607d" not found 29m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-45607d Scaled up replica set payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9 from 0 to 1 29m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9 Error creating: pods "payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-45607d: serviceaccount "payload-processing-e2e-ipp-b-45607d" not found 29m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Pulled pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 29m Normal Started pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Container started 29m Normal Started pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Container started 29m Normal Created pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Container created 29m Normal AddedInterface pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Add eth0 [10.134.1.28/23] from ovn-kubernetes 29m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Add eth0 [10.134.1.27/23] from ovn-kubernetes 29m Normal Created pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Container created 29m Normal Created pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Container created 29m Normal Pulled pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757 Created pod: e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb 29m Normal AddedInterface pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Add eth0 [10.133.0.157/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w0-ff4760-openshift-default Scaled up replica set e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757 from 1 to 2 29m Normal Started pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Container started 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w0-ff4760-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal Pulled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 29m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc Created pod: e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp 29m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf Created pod: e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l 29m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-262c10-openshift-default Scaled up replica set e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf from 1 to 2 29m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0dba31-openshift-default Scaled up replica set e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc from 1 to 2 29m Normal AddedInterface pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Add eth0 [10.132.0.203/23] from ovn-kubernetes 29m Normal Started pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Container started 29m Normal Created pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Container created 29m Normal Pulled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 29m Normal Created pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Container created 29m Normal AddedInterface pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Add eth0 [10.132.0.204/23] from ovn-kubernetes 29m Normal Started pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Container started 29m Warning ProbeError pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Startup probe error: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Startup probe failed: Get "http://10.133.0.157:15021/healthz/ready": dial tcp 10.133.0.157:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Startup probe error: Get "http://10.132.0.203:15021/healthz/ready": dial tcp 10.132.0.203:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Startup probe failed: Get "http://10.132.0.204:15021/healthz/ready": dial tcp 10.132.0.204:15021: connect: connection refused 29m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Startup probe error: Get "http://10.132.0.204:15021/healthz/ready": dial tcp 10.132.0.204:15021: connect: connection refused... 29m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Startup probe failed: Get "http://10.132.0.203:15021/healthz/ready": dial tcp 10.132.0.203:15021: connect: connection refused 29m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal Killing pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 Stopping container istio-proxy 28m Normal Killing pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-j6qzt Stopping container istio-proxy 28m Warning Unhealthy pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Readiness probe failed: HTTP probe failed with statuscode: 503 28m Warning ProbeError pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-z72k7 Readiness probe error: HTTP probe failed with statuscode: 503... 28m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw 28m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-35a94e-openshift-default Scaled up replica set e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc from 0 to 1 28m Normal AddedLabel gateway/e2e-shared-a-w1-35a94e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-35a94e 28m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Add eth0 [10.134.1.31/23] from ovn-kubernetes 28m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b Error creating: pods "payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e: serviceaccount "payload-processing-e2e-shared-a-w1-35a94e" not found 28m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b Created pod: payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd 28m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-35a94e Scaled up replica set payload-processing-e2e-shared-a-w1-35a94e-6996dd895b from 0 to 1 28m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-35a94e Scaled up replica set payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767 from 0 to 1 28m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767 Created pod: payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz 28m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-35a94e: serviceaccount "payload-processing-e2e-shared-a-w1-35a94e" not found 28m Normal Created pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Container created 28m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Container started 28m Normal Pulled pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Container created 28m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 28m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Add eth0 [10.134.1.32/23] from ovn-kubernetes 28m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Add eth0 [10.134.1.33/23] from ovn-kubernetes 28m Normal Started pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Container started 28m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 28m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 28m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 Created pod: e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx 28m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-dfb231-openshift-default Scaled up replica set e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 from 1 to 2 28m Normal AddedInterface pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Add eth0 [10.134.1.34/23] from ovn-kubernetes 28m Normal Created pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Container created 28m Normal Started pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Container started 28m Normal Pulled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Warning Unhealthy pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Startup probe failed: Get "http://10.134.1.34:15021/healthz/ready": dial tcp 10.134.1.34:15021: connect: connection refused 28m Warning ProbeError pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Startup probe error: Get "http://10.134.1.34:15021/healthz/ready": dial tcp 10.134.1.34:15021: connect: connection refused... 28m Normal Killing pod/payload-processing-e2e-ipp-a-60a531-764f46b8bd-s7rm2 Stopping container payload-processing 28m Normal Killing pod/payload-pre-processing-e2e-ipp-a-60a531-5b7d8f778c-t68n4 Stopping container payload-pre-processing 28m Normal Pulled pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 28m Normal Created pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Container created 28m Normal Started pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Container started 28m Warning ProbeError pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Startup probe error: Get "http://10.134.1.20:15021/healthz/ready": dial tcp 10.134.1.20:15021: connect: connection refused... 28m Warning Unhealthy pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Startup probe failed: Get "http://10.134.1.20:15021/healthz/ready": dial tcp 10.134.1.20:15021: connect: connection refused 27m Normal Killing pod/payload-pre-processing-e2e-ipp-b-45607d-6d95bcc9c9-h775m Stopping container payload-pre-processing 27m Normal Killing pod/payload-processing-e2e-ipp-b-45607d-c7bcdf84b-l9rf5 Stopping container payload-processing 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-60a531-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-60a531-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-60a531-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-60a531-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599 Created pod: payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599 Error creating: pods "payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-60a531: serviceaccount "payload-processing-e2e-ipp-a-60a531" not found 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-60a531-777f9cb8f8 Error creating: pods "payload-processing-e2e-ipp-a-60a531-777f9cb8f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-60a531: serviceaccount "payload-processing-e2e-ipp-a-60a531" not found 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-60a531 Scaled up replica set payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-60a531-777f9cb8f8 Created pod: payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-60a531 Scaled up replica set payload-processing-e2e-ipp-a-60a531-777f9cb8f8 from 0 to 1 27m Normal Created pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Container created 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Add eth0 [10.134.1.38/23] from ovn-kubernetes 27m Normal Pulled pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal Started pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Container started 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Add eth0 [10.132.0.208/23] from ovn-kubernetes 27m Normal Created pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Container created 27m Normal Started pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Container started 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Container started 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Container created 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Container created 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Add eth0 [10.133.0.159/23] from ovn-kubernetes 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Add eth0 [10.132.0.209/23] from ovn-kubernetes 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Container started 27m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled up replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 1 to 4 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Add eth0 [10.134.1.39/23] from ovn-kubernetes 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Container created 27m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Readiness probe failed: Get "http://10.134.1.26:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Readiness probe error: Get "http://10.134.1.26:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Container started 27m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Startup probe error: Get "http://10.133.0.159:15021/healthz/ready": dial tcp 10.133.0.159:15021: connect: connection refused... 27m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Startup probe error: Get "http://10.132.0.209:15021/healthz/ready": dial tcp 10.132.0.209:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Startup probe failed: Get "http://10.132.0.209:15021/healthz/ready": dial tcp 10.132.0.209:15021: connect: connection refused 27m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Startup probe failed: Get "http://10.133.0.159:15021/healthz/ready": dial tcp 10.133.0.159:15021: connect: connection refused 27m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-35a94e-openshift-default Scaled up replica set e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc from 1 to 4 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz 27m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 27m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Startup probe failed: Get "http://10.134.1.39:15021/healthz/ready": dial tcp 10.134.1.39:15021: connect: connection refused 27m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Container started 27m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Add eth0 [10.133.0.161/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Add eth0 [10.134.1.41/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Startup probe error: Get "http://10.134.1.39:15021/healthz/ready": dial tcp 10.134.1.39:15021: connect: connection refused... 27m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Container created 27m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Add eth0 [10.132.0.210/23] from ovn-kubernetes 27m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Container started 27m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Container created 27m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Container started 27m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Container created 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-ec92c7-openshift-default-5f75d64979 Created pod: e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 27m Normal ScalingReplicaSet deployment/e2e-ipp-b-ec92c7-openshift-default Scaled up replica set e2e-ipp-b-ec92c7-openshift-default-5f75d64979 from 0 to 1 27m Normal AddedLabel gateway/e2e-ipp-b-ec92c7 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-ec92c7 27m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Readiness probe error: Get "http://10.134.1.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 27m Normal AddedInterface pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Add eth0 [10.132.0.211/23] from ovn-kubernetes 27m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Readiness probe failed: Get "http://10.134.1.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Container started 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Container created 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled up replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 0 to 1 27m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled down replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 7 to 6 27m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Startup probe failed: Get "http://10.132.0.210:15021/healthz/ready": dial tcp 10.132.0.210:15021: connect: connection refused 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default New size: 6; reason: All metrics below target 27m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Deleted pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s 27m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Startup probe error: Get "http://10.132.0.210:15021/healthz/ready": dial tcp 10.132.0.210:15021: connect: connection refused... 27m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd 27m Normal AddedLabel gateway/e2e-shared-b-w1-9dc17a Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-9dc17a 27m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Startup probe failed: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused 27m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Startup probe error: Get "http://10.133.0.161:15021/healthz/ready": dial tcp 10.133.0.161:15021: connect: connection refused... 27m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 27m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6wt5s Stopping container istio-proxy 27m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Add eth0 [10.134.1.42/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Startup probe error: Get "http://10.134.1.41:15021/healthz/ready": dial tcp 10.134.1.41:15021: connect: connection refused... 27m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Startup probe failed: Get "http://10.134.1.41:15021/healthz/ready": dial tcp 10.134.1.41:15021: connect: connection refused 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-ec92c7 Scaled up replica set payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459 from 0 to 1 27m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Container started 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-ec92c7-5cb876776 Created pod: payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-ec92c7-5cb876776 Error creating: pods "payload-processing-e2e-ipp-b-ec92c7-5cb876776-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-ec92c7: serviceaccount "payload-processing-e2e-ipp-b-ec92c7" not found 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459 Created pod: payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w 27m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Container created 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459 Error creating: pods "payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-ec92c7: serviceaccount "payload-processing-e2e-ipp-b-ec92c7" not found 27m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-ec92c7 Scaled up replica set payload-processing-e2e-ipp-b-ec92c7-5cb876776 from 0 to 1 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Add eth0 [10.134.1.43/23] from ovn-kubernetes 27m Normal AddedInterface pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Add eth0 [10.134.1.44/23] from ovn-kubernetes 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Pulled pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Container created 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-45607d Scaled up replica set payload-processing-e2e-ipp-b-45607d-6b4677fbf6 from 0 to 1 27m Normal Created pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Container created 27m Normal Started pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Container started 27m Normal Started pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Container started 27m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Startup probe error: Get "http://10.134.1.26:15021/healthz/ready": dial tcp 10.134.1.26:15021: connect: connection refused... 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-45607d Scaled up replica set payload-pre-processing-e2e-ipp-b-45607d-76d84858b7 from 0 to 1 27m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Startup probe failed: Get "http://10.134.1.26:15021/healthz/ready": dial tcp 10.134.1.26:15021: connect: connection refused 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7 Error creating: pods "payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-45607d: serviceaccount "payload-processing-e2e-ipp-b-45607d" not found 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-45607d-6b4677fbf6 Created pod: payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7 Created pod: payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj 27m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-45607d-6b4677fbf6 Error creating: pods "payload-processing-e2e-ipp-b-45607d-6b4677fbf6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-45607d: serviceaccount "payload-processing-e2e-ipp-b-45607d" not found 27m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal Started pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Container started 27m Normal Created pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Container created 27m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-9dc17a Scaled up replica set payload-processing-e2e-shared-b-w1-9dc17a-c856945c8 from 0 to 1 27m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Add eth0 [10.134.1.45/23] from ovn-kubernetes 27m Normal Started pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Container started 27m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8 Error creating: pods "payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a: serviceaccount "payload-processing-e2e-shared-b-w1-9dc17a" not found 27m Normal Created pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Container created 27m Normal Pulled pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Add eth0 [10.134.1.46/23] from ovn-kubernetes 27m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-9dc17a Scaled up replica set payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79 from 0 to 1 27m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79 Created pod: payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb 27m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79 Error creating: pods "payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-9dc17a: serviceaccount "payload-processing-e2e-shared-b-w1-9dc17a" not found 27m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Add eth0 [10.134.1.47/23] from ovn-kubernetes 27m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8 Created pod: payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h 27m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Add eth0 [10.134.1.48/23] from ovn-kubernetes 27m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Startup probe error: Get "http://10.134.1.31:15021/healthz/ready": dial tcp 10.134.1.31:15021: connect: connection refused... 27m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Container created 27m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Container started 27m Normal Pulled pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 27m Normal Created pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Container created 27m Normal Started pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Container started 27m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Startup probe failed: Get "http://10.134.1.31:15021/healthz/ready": dial tcp 10.134.1.31:15021: connect: connection refused 27m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled up replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 4 to 8 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm 27m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default New size: 8; reason: 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Add eth0 [10.132.0.213/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Add eth0 [10.132.0.212/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Add eth0 [10.133.0.162/23] from ovn-kubernetes 27m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Add eth0 [10.134.1.49/23] from ovn-kubernetes 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 27m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Container started 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Container created 27m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Container created 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Container started 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Container started 27m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Container started 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Container created 27m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Container created 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Startup probe error: Get "http://10.132.0.212:15021/healthz/ready": dial tcp 10.132.0.212:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Startup probe failed: Get "http://10.132.0.212:15021/healthz/ready": dial tcp 10.132.0.212:15021: connect: connection refused 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Startup probe failed: Get "http://10.132.0.213:15021/healthz/ready": dial tcp 10.132.0.213:15021: connect: connection refused 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Startup probe error: Get "http://10.132.0.213:15021/healthz/ready": dial tcp 10.132.0.213:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Startup probe failed: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Startup probe error: Get "http://10.133.0.162:15021/healthz/ready": dial tcp 10.133.0.162:15021: connect: connection refused... 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Startup probe failed: Get "http://10.134.1.49:15021/healthz/ready": dial tcp 10.134.1.49:15021: connect: connection refused 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Startup probe error: Get "http://10.134.1.49:15021/healthz/ready": dial tcp 10.134.1.49:15021: connect: connection refused... 26m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd 26m Normal AddedLabel gateway/e2e-ipp-a-2d21ab Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-2d21ab 26m Normal ScalingReplicaSet deployment/e2e-ipp-a-2d21ab-openshift-default Scaled up replica set e2e-ipp-a-2d21ab-openshift-default-7d7974c69f from 0 to 1 26m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Add eth0 [10.134.1.51/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt 26m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled up replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 8 to 10 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default New size: 10; reason: 26m Normal SuccessfulCreate replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc (combined from similar events): Created pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg 26m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Container started 26m Normal Started pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Container started 26m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Add eth0 [10.133.0.163/23] from ovn-kubernetes 26m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Container created 26m Normal AddedInterface pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Add eth0 [10.132.0.214/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Container created 26m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-2d21ab Scaled up replica set payload-processing-e2e-ipp-a-2d21ab-698d76c964 from 0 to 1 26m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-2d21ab Scaled up replica set payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855 from 0 to 1 26m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855 Error creating: pods "payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-2d21ab: serviceaccount "payload-processing-e2e-ipp-a-2d21ab" not found 26m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855 Created pod: payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 26m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-2d21ab-698d76c964 Error creating: pods "payload-processing-e2e-ipp-a-2d21ab-698d76c964-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-2d21ab: serviceaccount "payload-processing-e2e-ipp-a-2d21ab" not found 26m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-2d21ab-698d76c964 Created pod: payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs 26m Normal Created pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Container created 26m Normal Started pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Container started 26m Normal Pulled pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal Started pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Container started 26m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 26m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Add eth0 [10.134.1.54/23] from ovn-kubernetes 26m Normal Created pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Container created 26m Normal AddedInterface pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Add eth0 [10.134.1.53/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Startup probe failed: Get "http://10.132.0.214:15021/healthz/ready": dial tcp 10.132.0.214:15021: connect: connection refused 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Startup probe error: Get "http://10.132.0.214:15021/healthz/ready": dial tcp 10.132.0.214:15021: connect: connection refused... 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Created pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh 26m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Startup probe error: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Startup probe failed: Get "http://10.133.0.163:15021/healthz/ready": dial tcp 10.133.0.163:15021: connect: connection refused 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default New size: 8; reason: All metrics below target 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-35a94e-openshift-default Scaled up replica set e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc from 4 to 8 26m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Add eth0 [10.133.0.165/23] from ovn-kubernetes 26m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Container started 26m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Add eth0 [10.132.0.215/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Container created 26m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Container created 26m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Add eth0 [10.133.0.164/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Container started 26m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Container started 26m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Container created 26m Normal AddedInterface pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Add eth0 [10.134.1.55/23] from ovn-kubernetes 26m Normal Started pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Container started 26m Normal Pulled pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Container created 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-ec92c7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-ec92c7-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled down replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 6 to 4 26m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Startup probe failed: Get "http://10.134.1.55:15021/healthz/ready": dial tcp 10.134.1.55:15021: connect: connection refused 26m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Startup probe failed: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused 26m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Startup probe error: Get "http://10.133.0.164:15021/healthz/ready": dial tcp 10.133.0.164:15021: connect: connection refused... 26m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr Stopping container istio-proxy 26m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Startup probe error: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused... 26m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Deleted pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-6q4jr 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default New size: 4; reason: All metrics below target 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-674082-openshift-default-577d994559 Created pod: e2e-shared-b-w4-674082-openshift-default-577d994559-89swj 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww Stopping container istio-proxy 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-674082-openshift-default Scaled up replica set e2e-shared-b-w4-674082-openshift-default-577d994559 from 1 to 2 26m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Startup probe failed: Get "http://10.132.0.215:15021/healthz/ready": dial tcp 10.132.0.215:15021: connect: connection refused 26m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Startup probe failed: Get "http://10.133.0.164:15021/healthz/ready": dial tcp 10.133.0.164:15021: connect: connection refused 26m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Startup probe error: Get "http://10.134.1.55:15021/healthz/ready": dial tcp 10.134.1.55:15021: connect: connection refused... 26m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Startup probe error: Get "http://10.132.0.215:15021/healthz/ready": dial tcp 10.132.0.215:15021: connect: connection refused... 26m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Deleted pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww 26m Normal Created pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Container created 26m Normal AddedInterface pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Add eth0 [10.132.0.216/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Started pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Container started 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w2-44bc69-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9 Created pod: e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-44bc69-openshift-default Scaled up replica set e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9 from 1 to 2 26m Normal Created pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Container created 26m Normal Pulled pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled up replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 3 to 4 26m Normal AddedInterface pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Add eth0 [10.134.1.57/23] from ovn-kubernetes 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss 26m Normal Started pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Container started 26m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Container started 26m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Add eth0 [10.133.0.166/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Container created 26m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Startup probe error: Get "http://10.132.0.216:15021/healthz/ready": dial tcp 10.132.0.216:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Startup probe failed: Get "http://10.132.0.216:15021/healthz/ready": dial tcp 10.132.0.216:15021: connect: connection refused 26m Normal SuccessfulCreate replicaset/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 Created pod: e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd 26m Warning ProbeError pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Startup probe error: Get "http://10.134.1.57:15021/healthz/ready": dial tcp 10.134.1.57:15021: connect: connection refused... 26m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-dfb231-openshift-default Scaled up replica set e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 from 2 to 3 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 26m Warning Unhealthy pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Startup probe failed: Get "http://10.134.1.57:15021/healthz/ready": dial tcp 10.134.1.57:15021: connect: connection refused 26m Normal Created pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Container created 26m Normal Started pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Container started 26m Normal AddedInterface pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Add eth0 [10.133.0.168/23] from ovn-kubernetes 26m Normal Pulled pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 26m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Startup probe error: Get "http://10.133.0.166:15021/healthz/ready": dial tcp 10.133.0.166:15021: connect: connection refused... 26m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Startup probe failed: Get "http://10.133.0.166:15021/healthz/ready": dial tcp 10.133.0.166:15021: connect: connection refused 26m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 26m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Startup probe error: Get "http://10.133.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 26m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Startup probe failed: Get "http://10.133.0.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Startup probe failed: Get "http://10.133.0.168:15021/healthz/ready": dial tcp 10.133.0.168:15021: connect: connection refused 26m Warning ProbeError pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Startup probe error: Get "http://10.133.0.168:15021/healthz/ready": dial tcp 10.133.0.168:15021: connect: connection refused... 26m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 Created pod: e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw 26m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 26m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-34722e-openshift-default Scaled up replica set e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 from 2 to 3 26m Normal AddedInterface pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Add eth0 [10.133.0.169/23] from ovn-kubernetes 26m Normal Created pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Container created 26m Normal Started pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Container started 26m Normal Pulled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning ProbeError pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Startup probe error: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Startup probe failed: Get "http://10.133.0.169:15021/healthz/ready": dial tcp 10.133.0.169:15021: connect: connection refused 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default New size: 3; reason: All metrics below target 25m Normal SuccessfulDelete replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Deleted pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph 25m Normal ScalingReplicaSet deployment/e2e-ipp-a-6b3855-openshift-default Scaled down replica set e2e-ipp-a-6b3855-openshift-default-5f877686d5 from 4 to 3 25m Normal Killing pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gdqph Stopping container istio-proxy 25m Normal Pulled pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Created pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Container created 25m Normal Started pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Container started 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled up replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 1 to 4 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 25m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Container created 25m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Container created 25m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Container started 25m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Add eth0 [10.133.0.171/23] from ovn-kubernetes 25m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Container started 25m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Add eth0 [10.133.0.170/23] from ovn-kubernetes 25m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Add eth0 [10.134.1.61/23] from ovn-kubernetes 25m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Warning Unhealthy pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Startup probe failed: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused 25m Warning ProbeError pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Startup probe error: Get "http://10.132.0.211:15021/healthz/ready": dial tcp 10.132.0.211:15021: connect: connection refused... 25m Normal Killing pod/payload-processing-e2e-ipp-b-ec92c7-5cb876776-fknkm Stopping container payload-processing 25m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Container created 25m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Container started 25m Normal Killing pod/payload-pre-processing-e2e-ipp-b-ec92c7-c5f4f6459-g4c6w Stopping container payload-pre-processing 25m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Readiness probe failed: Get "http://10.134.1.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 25m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Readiness probe error: Get "http://10.134.1.42:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 25m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Startup probe error: Get "http://10.133.0.170:15021/healthz/ready": dial tcp 10.133.0.170:15021: connect: connection refused... 25m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Startup probe error: Get "http://10.133.0.171:15021/healthz/ready": dial tcp 10.133.0.171:15021: connect: connection refused... 25m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Container started 25m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Startup probe failed: Get "http://10.133.0.170:15021/healthz/ready": dial tcp 10.133.0.170:15021: connect: connection refused 25m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Container created 25m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Startup probe failed: Get "http://10.133.0.171:15021/healthz/ready": dial tcp 10.133.0.171:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Startup probe error: Get "http://10.134.1.61:15021/healthz/ready": dial tcp 10.134.1.61:15021: connect: connection refused... 25m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Startup probe failed: Get "http://10.134.1.61:15021/healthz/ready": dial tcp 10.134.1.61:15021: connect: connection refused 25m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc Created pod: e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h 25m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0dba31-openshift-default Scaled up replica set e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc from 2 to 3 25m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 25m Normal Started pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Container started 25m Normal Created pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Container created 25m Normal Pulled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 25m Normal AddedInterface pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Add eth0 [10.133.0.172/23] from ovn-kubernetes 25m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Startup probe failed: Get "http://10.134.1.42:15021/healthz/ready": dial tcp 10.134.1.42:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Startup probe error: Get "http://10.134.1.42:15021/healthz/ready": dial tcp 10.134.1.42:15021: connect: connection refused... 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 25m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Startup probe failed: Get "http://10.133.0.172:15021/healthz/ready": dial tcp 10.133.0.172:15021: connect: connection refused 25m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Startup probe error: Get "http://10.133.0.172:15021/healthz/ready": dial tcp 10.133.0.172:15021: connect: connection refused... 25m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 25m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Normal Killing pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Stopping container payload-pre-processing 24m Normal AddedLabel gateway/e2e-ipp-a-02cffa Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-02cffa 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-02cffa-openshift-default Scaled up replica set e2e-ipp-a-02cffa-openshift-default-6644dbdd6c from 0 to 1 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz 24m Normal Killing pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Stopping container payload-processing 24m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Add eth0 [10.134.1.64/23] from ovn-kubernetes 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-ec92c7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-ec92c7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-02cffa-555549cbfb Created pod: payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-02cffa Scaled up replica set payload-processing-e2e-ipp-a-02cffa-555549cbfb from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f Created pod: payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-02cffa-555549cbfb Error creating: pods "payload-processing-e2e-ipp-a-02cffa-555549cbfb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-02cffa: serviceaccount "payload-processing-e2e-ipp-a-02cffa" not found 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-02cffa Scaled up replica set payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f from 0 to 1 24m Warning Unhealthy pod/payload-pre-processing-e2e-ipp-a-2d21ab-7d87cd7855-hj9l9 Readiness probe failed: timeout: failed to connect service "10.134.1.54:50052" within 1s: context deadline exceeded 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f Error creating: pods "payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-02cffa: serviceaccount "payload-processing-e2e-ipp-a-02cffa" not found 24m Warning Unhealthy pod/payload-processing-e2e-ipp-a-2d21ab-698d76c964-nkgrs Readiness probe failed: timeout: failed to connect service "10.134.1.53:50052" within 1s: context deadline exceeded 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default New size: 8; reason: All metrics below target 24m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt 24m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Add eth0 [10.133.0.184/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 24m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled up replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 4 to 8 24m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Add eth0 [10.133.0.183/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh 24m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Container created 24m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Add eth0 [10.134.1.85/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Add eth0 [10.134.1.83/23] from ovn-kubernetes 24m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Container created 24m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Add eth0 [10.134.1.84/23] from ovn-kubernetes 24m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Container started 24m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Container started 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Add eth0 [10.134.1.82/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Pulled pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Container created 24m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Container started 24m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Container created 24m Normal Created pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Container created 24m Normal Started pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Container started 24m Normal Created pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Container created 24m Normal Started pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Container started 24m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Container started 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Startup probe failed: Get "http://10.133.0.183:15021/healthz/ready": dial tcp 10.133.0.183:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Startup probe error: Get "http://10.133.0.183:15021/healthz/ready": dial tcp 10.133.0.183:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Startup probe error: Get "http://10.133.0.184:15021/healthz/ready": dial tcp 10.133.0.184:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Startup probe failed: Get "http://10.133.0.184:15021/healthz/ready": dial tcp 10.133.0.184:15021: connect: connection refused 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-ec92c7 Scaled up replica set payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467 from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-ec92c7 Scaled up replica set payload-processing-e2e-ipp-b-ec92c7-b5466f7c8 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467 Created pod: payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8 Error creating: pods "payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-ec92c7: serviceaccount "payload-processing-e2e-ipp-b-ec92c7" not found 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467 Error creating: pods "payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-ec92c7: serviceaccount "payload-processing-e2e-ipp-b-ec92c7" not found 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8 Created pod: payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Add eth0 [10.132.0.220/23] from ovn-kubernetes 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Startup probe failed: Get "http://10.134.1.85:15021/healthz/ready": dial tcp 10.134.1.85:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Startup probe error: Get "http://10.134.1.84:15021/healthz/ready": dial tcp 10.134.1.84:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Startup probe error: Get "http://10.134.1.85:15021/healthz/ready": dial tcp 10.134.1.85:15021: connect: connection refused... 24m Normal Started pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Container started 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Container created 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Startup probe failed: Get "http://10.134.1.84:15021/healthz/ready": dial tcp 10.134.1.84:15021: connect: connection refused 24m Normal AddedInterface pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Add eth0 [10.134.1.87/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Container created 24m Normal Started pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Container started 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-2d21ab-openshift-default Scaled up replica set e2e-ipp-a-2d21ab-openshift-default-7d7974c69f from 1 to 4 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 24m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Container created 24m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Container started 24m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Add eth0 [10.134.1.88/23] from ovn-kubernetes 24m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Add eth0 [10.133.0.185/23] from ovn-kubernetes 24m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Stopping container istio-proxy 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w6-a914e9-openshift-default New size: 3; reason: All metrics below target 24m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Container created 24m Normal ScalingReplicaSet deployment/e2e-shared-b-w6-a914e9-openshift-default Scaled down replica set e2e-shared-b-w6-a914e9-openshift-default-7997967f6c from 4 to 3 24m Normal SuccessfulDelete replicaset/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c Deleted pod: e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td 24m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Container started 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-52g24 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-6b3855-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-6b3855-openshift-default Scaled up replica set e2e-ipp-a-6b3855-openshift-default-5f877686d5 from 3 to 5 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b 24m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Readiness probe error: Get "http://10.134.1.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Readiness probe failed: Get "http://10.134.1.51:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal AddedInterface pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Add eth0 [10.133.0.186/23] from ovn-kubernetes 24m Normal Started pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Container started 24m Normal Created pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Container created 24m Normal Pulled pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-bx4nb 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq 24m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Readiness probe error: Get "http://10.134.0.229:15021/healthz/ready": dial tcp 10.134.0.229:15021: connect: connection refused... 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc Created pod: e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c 24m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 3; reason: cpu resource utilization (percentage of request) above target 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-6b3855-openshift-default-5f877686d5 Created pod: e2e-ipp-a-6b3855-openshift-default-5f877686d5-mpg5z 24m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-nv9td Readiness probe failed: Get "http://10.134.0.229:15021/healthz/ready": dial tcp 10.134.0.229:15021: connect: connection refused 24m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Startup probe error: Get "http://10.133.0.185:15021/healthz/ready": dial tcp 10.133.0.185:15021: connect: connection refused... 24m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 3 24m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Stopping container istio-proxy 24m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Startup probe failed: Get "http://10.133.0.185:15021/healthz/ready": dial tcp 10.133.0.185:15021: connect: connection refused 24m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h Stopping container istio-proxy 24m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Stopping container istio-proxy 24m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Startup probe failed: Get "http://10.134.1.88:15021/healthz/ready": dial tcp 10.134.1.88:15021: connect: connection refused 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Startup probe error: Get "http://10.134.1.88:15021/healthz/ready": dial tcp 10.134.1.88:15021: connect: connection refused... 24m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled up replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 4 to 6 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz 24m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Created pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs 24m Normal Killing pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Stopping container istio-proxy 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 24m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Startup probe failed: Get "http://10.133.0.186:15021/healthz/ready": dial tcp 10.133.0.186:15021: connect: connection refused 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 24m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b Startup probe error: Get "http://10.133.0.186:15021/healthz/ready": dial tcp 10.133.0.186:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 Readiness probe error: HTTP probe failed with statuscode: 503... 24m Normal Created pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Container created 24m Normal Started pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Container started 24m Normal AddedInterface pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Add eth0 [10.134.1.89/23] from ovn-kubernetes 24m Normal Pulled pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Container started 24m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Add eth0 [10.134.1.90/23] from ovn-kubernetes 24m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Container created 24m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv 24m Normal ScalingReplicaSet deployment/e2e-ipp-a-2d21ab-openshift-default Scaled up replica set e2e-ipp-a-2d21ab-openshift-default-7d7974c69f from 4 to 8 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 24m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz 24m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default New size: 8; reason: 24m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 24m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Container created 24m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 24m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Add eth0 [10.133.0.187/23] from ovn-kubernetes 24m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Container created 24m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Container started 24m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Container started 24m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Startup probe failed: Get "http://10.134.1.90:15021/healthz/ready": dial tcp 10.134.1.90:15021: connect: connection refused 24m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Startup probe error: Get "http://10.134.1.89:15021/healthz/ready": dial tcp 10.134.1.89:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Startup probe error: Get "http://10.134.1.90:15021/healthz/ready": dial tcp 10.134.1.90:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Startup probe failed: Get "http://10.134.1.89:15021/healthz/ready": dial tcp 10.134.1.89:15021: connect: connection refused 24m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Readiness probe error: HTTP probe failed with statuscode: 503... 24m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Startup probe error: Get "http://10.134.1.51:15021/healthz/ready": dial tcp 10.134.1.51:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Startup probe failed: Get "http://10.134.1.51:15021/healthz/ready": dial tcp 10.134.1.51:15021: connect: connection refused 24m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb Error creating: pods "payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-2d21ab: serviceaccount "payload-processing-e2e-ipp-a-2d21ab" not found 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-2d21ab Scaled up replica set payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-2d21ab Scaled up replica set payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb Created pod: payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f Created pod: payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f Error creating: pods "payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-2d21ab: serviceaccount "payload-processing-e2e-ipp-a-2d21ab" not found 24m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Add eth0 [10.134.1.93/23] from ovn-kubernetes 24m Normal Started pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Container started 24m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 24m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 24m Normal Created pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Container created 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default New size: 9; reason: All metrics below target 23m Normal SuccessfulCreate replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f (combined from similar events): Created pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt 23m Normal ScalingReplicaSet deployment/e2e-ipp-a-2d21ab-openshift-default Scaled up replica set e2e-ipp-a-2d21ab-openshift-default-7d7974c69f from 8 to 9 23m Normal Killing pod/payload-pre-processing-e2e-ipp-a-6b3855-59f6844f87-djpsr Stopping container payload-pre-processing 23m Normal Killing pod/payload-processing-e2e-ipp-a-6b3855-555c7f6cbd-cjlds Stopping container payload-processing 23m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Add eth0 [10.132.0.225/23] from ovn-kubernetes 23m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Container started 23m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Container created 23m Normal Killing pod/payload-processing-e2e-ipp-a-60a531-777f9cb8f8-qzkpv Stopping container payload-processing 23m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Killing pod/payload-pre-processing-e2e-ipp-a-60a531-6bc8dfb599-wmbl4 Stopping container payload-pre-processing 23m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Container started 23m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal AddedInterface pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Add eth0 [10.134.1.97/23] from ovn-kubernetes 23m Normal Pulled pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 23m Normal Created pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Container created 23m Normal Started pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Container started 23m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Container started 23m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Add eth0 [10.132.0.227/23] from ovn-kubernetes 23m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Add eth0 [10.134.1.98/23] from ovn-kubernetes 23m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Container created 23m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Container created 23m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 23m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-674082-openshift-default Scaled up replica set e2e-shared-b-w4-674082-openshift-default-577d994559 from 2 to 3 23m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-674082-openshift-default-577d994559 Created pod: e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w 23m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Startup probe error: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Startup probe failed: Get "http://10.132.0.225:15021/healthz/ready": dial tcp 10.132.0.225:15021: connect: connection refused 23m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Startup probe failed: Get "http://10.132.0.227:15021/healthz/ready": dial tcp 10.132.0.227:15021: connect: connection refused 23m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Startup probe error: Get "http://10.132.0.227:15021/healthz/ready": dial tcp 10.132.0.227:15021: connect: connection refused... 23m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Startup probe error: Get "http://10.134.1.98:15021/healthz/ready": dial tcp 10.134.1.98:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Startup probe failed: Get "http://10.134.1.98:15021/healthz/ready": dial tcp 10.134.1.98:15021: connect: connection refused 23m Normal Killing pod/e2e-ipp-a-60a531-openshift-default-6c44b77d55-hl9k7 Stopping container istio-proxy 23m Normal Created pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Container created 23m Normal Started pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Container started 23m Normal AddedInterface pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Add eth0 [10.134.1.101/23] from ovn-kubernetes 23m Normal Pulled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 23m Normal SuccessfulCreate replicaset/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9 Created pod: e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 23m Normal ScalingReplicaSet deployment/e2e-ipp-b-b5b27e-openshift-default Scaled up replica set e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9 from 0 to 1 23m Normal AddedLabel gateway/e2e-ipp-b-b5b27e Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-b5b27e 22m Normal Killing pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Stopping container payload-pre-processing 22m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Startup probe failed: Get "http://10.134.1.101:15021/healthz/ready": dial tcp 10.134.1.101:15021: connect: connection refused 22m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Startup probe error: Get "http://10.134.1.101:15021/healthz/ready": dial tcp 10.134.1.101:15021: connect: connection refused... 22m Normal AddedLabel gateway/e2e-ipp-a-ec1091 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-ec1091 22m Normal ScalingReplicaSet deployment/e2e-ipp-a-ec1091-openshift-default Scaled up replica set e2e-ipp-a-ec1091-openshift-default-5b9f98669 from 0 to 1 22m Normal SuccessfulCreate replicaset/e2e-ipp-a-ec1091-openshift-default-5b9f98669 Created pod: e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq 22m Normal Killing pod/payload-pre-processing-e2e-ipp-b-ec92c7-86d548d467-fw2rs Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-ipp-b-ec92c7-b5466f7c8-4nbb5 Stopping container payload-processing 22m Warning Unhealthy pod/payload-pre-processing-e2e-ipp-a-02cffa-64cc65779f-qtckv Readiness probe failed: timeout: failed to connect service "10.134.1.82:50052" within 1s: context deadline exceeded 22m Normal AddedInterface pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Add eth0 [10.132.0.228/23] from ovn-kubernetes 22m Warning Unhealthy pod/payload-processing-e2e-ipp-a-02cffa-555549cbfb-86fwr Readiness probe failed: timeout: failed to connect service "10.134.1.83:50052" within 1s: context deadline exceeded 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-b5b27e Scaled up replica set payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79 from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79 Error creating: pods "payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-b5b27e: serviceaccount "payload-processing-e2e-ipp-b-b5b27e" not found 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-b5b27e Scaled up replica set payload-processing-e2e-ipp-b-b5b27e-d75dcfb from 0 to 1 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-b5b27e-d75dcfb Error creating: pods "payload-processing-e2e-ipp-b-b5b27e-d75dcfb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-b5b27e: serviceaccount "payload-processing-e2e-ipp-b-b5b27e" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79 Created pod: payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-b5b27e-d75dcfb Created pod: payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 22m Normal Pulled pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Add eth0 [10.133.0.188/23] from ovn-kubernetes 22m Normal Started pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Container started 22m Normal Started pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Container started 22m Normal Created pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Container created 22m Normal Created pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Container created 22m Normal AddedInterface pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Add eth0 [10.132.0.229/23] from ovn-kubernetes 22m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 22m Normal Killing pod/e2e-ipp-b-ec92c7-openshift-default-5f75d64979-xpp24 Stopping container istio-proxy 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 22m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 22m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v 22m Normal ScalingReplicaSet deployment/e2e-ipp-a-02cffa-openshift-default Scaled up replica set e2e-ipp-a-02cffa-openshift-default-6644dbdd6c from 1 to 4 22m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 22m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Add eth0 [10.134.1.103/23] from ovn-kubernetes 22m Normal AddedInterface pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Add eth0 [10.134.1.102/23] from ovn-kubernetes 22m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Container started 22m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Container created 22m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Readiness probe error: Get "http://10.134.1.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 22m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Add eth0 [10.132.0.230/23] from ovn-kubernetes 22m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Readiness probe failed: Get "http://10.134.1.64:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 22m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Container started 22m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Container created 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7 Error creating: pods "payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ec1091: serviceaccount "payload-processing-e2e-ipp-a-ec1091" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7 Created pod: payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ec1091 Scaled up replica set payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7 from 0 to 1 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ec1091-df8c5654 Created pod: payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ec1091 Scaled up replica set payload-processing-e2e-ipp-a-ec1091-df8c5654 from 0 to 1 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ec1091-df8c5654 Error creating: pods "payload-processing-e2e-ipp-a-ec1091-df8c5654-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ec1091: serviceaccount "payload-processing-e2e-ipp-a-ec1091" not found 22m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Startup probe failed: Get "http://10.132.0.230:15021/healthz/ready": dial tcp 10.132.0.230:15021: connect: connection refused 22m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Startup probe error: Get "http://10.132.0.230:15021/healthz/ready": dial tcp 10.132.0.230:15021: connect: connection refused... 22m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg Stopping container istio-proxy 22m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm Stopping container istio-proxy 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default New size: 8; reason: All metrics below target 22m Normal Killing pod/payload-pre-processing-e2e-shared-a-w3-097d90-78fbb47d67-pmrs7 Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-shared-a-w3-097d90-6f4f7d96cb-wvzt2 Stopping container payload-processing 22m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled down replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 10 to 8 22m Normal SuccessfulDelete replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Deleted pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-przjg 22m Normal SuccessfulDelete replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Deleted pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-kx8sm 22m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Startup probe failed: Get "http://10.134.1.103:15021/healthz/ready": dial tcp 10.134.1.103:15021: connect: connection refused 22m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Startup probe error: Get "http://10.134.1.103:15021/healthz/ready": dial tcp 10.134.1.103:15021: connect: connection refused... 22m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Container started 22m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Container created 22m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Container created 22m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Container created 22m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Container started 22m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Add eth0 [10.132.0.231/23] from ovn-kubernetes 22m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Add eth0 [10.132.0.232/23] from ovn-kubernetes 22m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Add eth0 [10.132.0.234/23] from ovn-kubernetes 22m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Container started 22m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 22m Normal Pulled pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Add eth0 [10.132.0.233/23] from ovn-kubernetes 22m Normal Created pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Container created 22m Normal Created pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Container created 22m Normal Started pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Container started 22m Normal Started pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Container started 22m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Startup probe error: Get "http://10.134.1.64:15021/healthz/ready": dial tcp 10.134.1.64:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Startup probe failed: Get "http://10.134.1.64:15021/healthz/ready": dial tcp 10.134.1.64:15021: connect: connection refused 22m Normal Killing pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm Stopping container istio-proxy 22m Normal Killing pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Stopping container istio-proxy 22m Warning ProbeError pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning Unhealthy pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-5d992 Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal Killing pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-6n27l Stopping container istio-proxy 22m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Container created 22m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Add eth0 [10.132.0.235/23] from ovn-kubernetes 22m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Container started 22m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Startup probe error: Get "http://10.132.0.231:15021/healthz/ready": dial tcp 10.132.0.231:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Startup probe failed: Get "http://10.132.0.234:15021/healthz/ready": dial tcp 10.132.0.234:15021: connect: connection refused 22m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Startup probe error: Get "http://10.132.0.234:15021/healthz/ready": dial tcp 10.132.0.234:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Startup probe failed: Get "http://10.132.0.231:15021/healthz/ready": dial tcp 10.132.0.231:15021: connect: connection refused 22m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Container created 22m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Add eth0 [10.134.1.104/23] from ovn-kubernetes 22m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Container started 22m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Add eth0 [10.132.0.236/23] from ovn-kubernetes 22m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 22m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Container started 22m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal SuccessfulCreate replicaset/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 Created pod: e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m 22m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Container created 22m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-34722e-openshift-default Scaled up replica set e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 from 3 to 4 22m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Container created 22m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Container started 22m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Add eth0 [10.133.0.190/23] from ovn-kubernetes 22m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Startup probe error: Get "http://10.132.0.235:15021/healthz/ready": dial tcp 10.132.0.235:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Startup probe failed: Get "http://10.132.0.235:15021/healthz/ready": dial tcp 10.132.0.235:15021: connect: connection refused 22m Normal Killing pod/e2e-shared-a-w3-097d90-openshift-default-64f69f59b7-p2hms Stopping container istio-proxy 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 22m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 22m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Startup probe error: Get "http://10.132.0.236:15021/healthz/ready": dial tcp 10.132.0.236:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Startup probe failed: Get "http://10.134.1.104:15021/healthz/ready": dial tcp 10.134.1.104:15021: connect: connection refused 22m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Startup probe failed: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused 22m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Startup probe error: Get "http://10.134.1.104:15021/healthz/ready": dial tcp 10.134.1.104:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Startup probe failed: Get "http://10.132.0.236:15021/healthz/ready": dial tcp 10.132.0.236:15021: connect: connection refused 22m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Startup probe error: Get "http://10.133.0.190:15021/healthz/ready": dial tcp 10.133.0.190:15021: connect: connection refused... 22m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-02cffa-858d55bbb Error creating: pods "payload-processing-e2e-ipp-a-02cffa-858d55bbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-02cffa: serviceaccount "payload-processing-e2e-ipp-a-02cffa" not found 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-02cffa Scaled up replica set payload-processing-e2e-ipp-a-02cffa-858d55bbb from 0 to 1 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-02cffa Scaled up replica set payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff Error creating: pods "payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-02cffa: serviceaccount "payload-processing-e2e-ipp-a-02cffa" not found 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff Created pod: payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-02cffa-858d55bbb Created pod: payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w 22m Normal Created pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Container created 22m Normal Started pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Container started 22m Normal Pulled pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 22m Normal AddedInterface pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Add eth0 [10.132.0.238/23] from ovn-kubernetes 22m Normal Killing pod/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-shared-b-w3-e15c99-55c549f87f-7rngf Stopping container payload-pre-processing 21m Warning ProbeError pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Startup probe error: Get "http://10.132.0.238:15021/healthz/ready": dial tcp 10.132.0.238:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Startup probe failed: Get "http://10.132.0.238:15021/healthz/ready": dial tcp 10.132.0.238:15021: connect: connection refused 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k 21m Normal Pulling pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 21m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Add eth0 [10.133.0.192/23] from ovn-kubernetes 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c Created pod: e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-02cffa-openshift-default New size: 7; reason: All metrics below target 21m Normal ScalingReplicaSet deployment/e2e-ipp-a-02cffa-openshift-default Scaled up replica set e2e-ipp-a-02cffa-openshift-default-6644dbdd6c from 4 to 7 21m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.319s (1.319s including waiting). Image size: 166612641 bytes. 21m Normal Created pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Container created 21m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Add eth0 [10.132.0.239/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Started pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Container started 21m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Container created 21m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Container started 21m Normal Killing pod/payload-processing-e2e-ipp-b-45607d-6b4677fbf6-d978h Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-ipp-b-45607d-76d84858b7-v2xnj Stopping container payload-pre-processing 21m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Container created 21m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Add eth0 [10.133.0.193/23] from ovn-kubernetes 21m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Container started 21m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Startup probe error: Get "http://10.132.0.239:15021/healthz/ready": dial tcp 10.132.0.239:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Startup probe failed: Get "http://10.132.0.239:15021/healthz/ready": dial tcp 10.132.0.239:15021: connect: connection refused 21m Normal ScalingReplicaSet deployment/e2e-ipp-b-45607d-openshift-default Scaled down replica set e2e-ipp-b-45607d-openshift-default-5988476fcc from 8 to 6 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt Stopping container istio-proxy 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-b-45607d-openshift-default New size: 6; reason: All metrics below target 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d Stopping container istio-proxy 21m Normal SuccessfulDelete replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Deleted pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d 21m Normal SuccessfulDelete replicaset/e2e-ipp-b-45607d-openshift-default-5988476fcc Deleted pod: e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt 21m Normal Killing pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h Stopping container istio-proxy 21m Normal Killing pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Stopping container istio-proxy 21m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Startup probe error: Get "http://10.133.0.193:15021/healthz/ready": dial tcp 10.133.0.193:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Startup probe failed: Get "http://10.133.0.193:15021/healthz/ready": dial tcp 10.133.0.193:15021: connect: connection refused 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-7qwxm Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Stopping container istio-proxy 21m Normal Killing pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Stopping container istio-proxy 21m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-4szwt Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Created pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Container created 21m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-2pd2n Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Started pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Container started 21m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Readiness probe failed: HTTP probe failed with statuscode: 503 21m Warning Unhealthy pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Created pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Container created 21m Warning ProbeError pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-mxnw7 Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-f4xw2 Readiness probe error: HTTP probe failed with statuscode: 503... 21m Normal AddedInterface pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Add eth0 [10.133.0.195/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal Pulled pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 21m Normal Started pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Container started 21m Normal AddedInterface pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Add eth0 [10.133.0.194/23] from ovn-kubernetes 21m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 Readiness probe error: HTTP probe failed with statuscode: 503... 21m Warning ProbeError pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Readiness probe error: HTTP probe failed with statuscode: 503... 21m Normal AddedInterface pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Add eth0 [10.133.0.196/23] from ovn-kubernetes 21m Warning Unhealthy pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-tk9bd Readiness probe failed: HTTP probe failed with statuscode: 503 21m Normal Created pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Container created 21m Normal Started pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Container started 21m Normal Pulled pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Add eth0 [10.132.0.241/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 21m Normal Killing pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Stopping container payload-pre-processing 21m Normal Created pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Container created 21m Normal Started pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Container started 21m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-262c10-openshift-default Scaled up replica set e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf from 2 to 3 21m Normal Killing pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Stopping container payload-processing 21m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf Created pod: e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 21m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Startup probe error: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Startup probe failed: Get "http://10.133.0.194:15021/healthz/ready": dial tcp 10.133.0.194:15021: connect: connection refused 21m Normal Started pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Container started 21m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 (combined from similar events): Created pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 21m Normal Created pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Container created 21m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled up replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 8 to 10 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 21m Normal AddedInterface pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Add eth0 [10.133.0.197/23] from ovn-kubernetes 21m Normal Pulled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Add eth0 [10.134.1.110/23] from ovn-kubernetes 21m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Container started 21m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Container created 21m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Startup probe error: Get "http://10.133.0.196:15021/healthz/ready": dial tcp 10.133.0.196:15021: connect: connection refused... 21m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Startup probe failed: Get "http://10.133.0.196:15021/healthz/ready": dial tcp 10.133.0.196:15021: connect: connection refused 21m Normal Pulled pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Add eth0 [10.132.0.242/23] from ovn-kubernetes 21m Normal Created pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Container created 21m Normal Started pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Container started 21m Normal Killing pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Stopping container payload-processing 21m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Startup probe failed: Get "http://10.132.0.241:15021/healthz/ready": dial tcp 10.132.0.241:15021: connect: connection refused 21m Normal Killing pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Stopping container payload-pre-processing 21m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Startup probe error: Get "http://10.132.0.241:15021/healthz/ready": dial tcp 10.132.0.241:15021: connect: connection refused... 21m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Startup probe error: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Startup probe failed: Get "http://10.133.0.197:15021/healthz/ready": dial tcp 10.133.0.197:15021: connect: connection refused 21m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Startup probe failed: Get "http://10.134.1.110:15021/healthz/ready": dial tcp 10.134.1.110:15021: connect: connection refused 21m Normal SuccessfulCreate replicaset/e2e-ipp-b-4bc875-openshift-default-5798f46f5f Created pod: e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg 21m Normal ScalingReplicaSet deployment/e2e-ipp-b-4bc875-openshift-default Scaled up replica set e2e-ipp-b-4bc875-openshift-default-5798f46f5f from 0 to 1 21m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Startup probe error: Get "http://10.134.1.110:15021/healthz/ready": dial tcp 10.134.1.110:15021: connect: connection refused... 21m Normal AddedLabel gateway/e2e-ipp-b-4bc875 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-4bc875 21m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Startup probe failed: Get "http://10.132.0.242:15021/healthz/ready": dial tcp 10.132.0.242:15021: connect: connection refused 21m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Startup probe error: Get "http://10.132.0.242:15021/healthz/ready": dial tcp 10.132.0.242:15021: connect: connection refused... 21m Normal AddedInterface pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Add eth0 [10.132.0.243/23] from ovn-kubernetes 21m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w0-dfb231-openshift-default New size: 2; reason: All metrics below target 21m Normal Killing pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx Stopping container istio-proxy 21m Normal ScalingReplicaSet deployment/e2e-shared-b-w0-dfb231-openshift-default Scaled down replica set e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 from 3 to 2 21m Normal SuccessfulDelete replicaset/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5 Deleted pod: e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-qr4qx 21m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-4bc875 Scaled up replica set payload-processing-e2e-ipp-b-4bc875-765d5b6fcb from 0 to 1 21m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb Error creating: pods "payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-4bc875: serviceaccount "payload-processing-e2e-ipp-b-4bc875" not found 21m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-4bc875 Scaled up replica set payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf from 0 to 1 21m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf Created pod: payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 21m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb Created pod: payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 21m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf Error creating: pods "payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-4bc875: serviceaccount "payload-processing-e2e-ipp-b-4bc875" not found 21m Normal Killing pod/e2e-shared-a-w6-2f6fed-openshift-default-6f5bcb6c89-vp2jt Stopping container istio-proxy 21m Warning Unhealthy pod/payload-processing-e2e-worker-w3-889ca9-7dcc755ffb-2d9dj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-889ca9-958dd6d55-rfmrk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Add eth0 [10.134.1.112/23] from ovn-kubernetes 21m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 21m Normal Created pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Container created 21m Normal Started pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Container started 21m Normal Pulled pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Add eth0 [10.134.1.113/23] from ovn-kubernetes 21m Normal Started pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Container started 21m Normal Created pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Container created 21m Warning Unhealthy pod/payload-processing-e2e-shared-a-w6-2f6fed-76f888db46-xr9t7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal SuccessfulCreate replicaset/e2e-ipp-a-7226a8-openshift-default-bf9bc5767 Created pod: e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w 21m Normal AddedLabel gateway/e2e-ipp-a-7226a8 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-7226a8 21m Normal ScalingReplicaSet deployment/e2e-ipp-a-7226a8-openshift-default Scaled up replica set e2e-ipp-a-7226a8-openshift-default-bf9bc5767 from 0 to 1 21m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w6-2f6fed-6d65c87b5f-4d2r2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 21m Normal AddedInterface pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Add eth0 [10.134.1.115/23] from ovn-kubernetes 21m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5 Error creating: pods "payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-7226a8: serviceaccount "payload-processing-e2e-ipp-a-7226a8" not found 21m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-7226a8 Scaled up replica set payload-processing-e2e-ipp-a-7226a8-68d55987b8 from 0 to 1 21m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-7226a8-68d55987b8 Created pod: payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 21m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-7226a8 Scaled up replica set payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5 from 0 to 1 21m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5 Created pod: payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn 21m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-7226a8-68d55987b8 Error creating: pods "payload-processing-e2e-ipp-a-7226a8-68d55987b8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-7226a8: serviceaccount "payload-processing-e2e-ipp-a-7226a8" not found 21m Normal Started pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Container started 21m Normal Created pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Container created 21m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Add eth0 [10.134.1.116/23] from ovn-kubernetes 21m Normal AddedInterface pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Add eth0 [10.134.1.117/23] from ovn-kubernetes 21m Normal Pulled pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 21m Normal Started pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Container started 21m Normal Created pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Container created 21m Normal Killing pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Stopping container payload-pre-processing 21m Normal Killing pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Stopping container payload-processing 20m Normal Pulled pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Started pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Container started 20m Normal Created pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Container created 20m Normal Killing pod/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-ipp-b-b5b27e-d75dcfb-7svt8 Stopping container payload-processing 20m Normal Created pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Container created 20m Normal Pulled pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal Started pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Container started 20m Warning Unhealthy pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Startup probe failed: Get "http://10.132.0.228:15021/healthz/ready": dial tcp 10.132.0.228:15021: connect: connection refused 20m Warning ProbeError pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Startup probe error: Get "http://10.132.0.228:15021/healthz/ready": dial tcp 10.132.0.228:15021: connect: connection refused... 20m Warning ProbeError pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Startup probe error: Get "http://10.134.1.102:15021/healthz/ready": dial tcp 10.134.1.102:15021: connect: connection refused... 20m Warning Unhealthy pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Startup probe failed: Get "http://10.134.1.102:15021/healthz/ready": dial tcp 10.134.1.102:15021: connect: connection refused 20m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w6-a914e9-58f9d696b7-64j9z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Warning Unhealthy pod/payload-processing-e2e-shared-b-w6-a914e9-6587fcffdd-w9cqm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 20m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ec1091-556ff68d7-bz48w Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-ipp-a-ec1091-df8c5654-6wjxm Stopping container payload-processing 20m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-b5b27e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 20m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-b5b27e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 20m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-ec1091-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 20m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-ec1091-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 20m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Stopping container istio-proxy 20m Normal Killing pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Stopping container istio-proxy 20m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-jlpp6 Readiness probe error: HTTP probe failed with statuscode: 503... 20m Warning Unhealthy pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Readiness probe failed: HTTP probe failed with statuscode: 503 20m Warning ProbeError pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-9l9bc Readiness probe error: HTTP probe failed with statuscode: 503... 20m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-b5b27e Scaled up replica set payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf from 0 to 1 20m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-b5b27e Scaled up replica set payload-processing-e2e-ipp-b-b5b27e-65576cb866 from 0 to 1 20m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-b5b27e-65576cb866 Created pod: payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p 20m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf Created pod: payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk 20m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf Error creating: pods "payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-b5b27e: serviceaccount "payload-processing-e2e-ipp-b-b5b27e" not found 20m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-b5b27e-65576cb866 Error creating: pods "payload-processing-e2e-ipp-b-b5b27e-65576cb866-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-b5b27e: serviceaccount "payload-processing-e2e-ipp-b-b5b27e" not found 20m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal Started pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Container started 20m Normal AddedInterface pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Add eth0 [10.132.0.247/23] from ovn-kubernetes 20m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Add eth0 [10.132.0.246/23] from ovn-kubernetes 20m Normal Started pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Container started 20m Normal Pulled pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 20m Normal Created pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Container created 20m Normal Created pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Container created 20m Normal ScalingReplicaSet deployment/e2e-shared-a-w4-262c10-openshift-default Scaled up replica set e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf from 3 to 4 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w4-262c10-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 20m Normal Killing pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Stopping container payload-pre-processing 20m Normal SuccessfulCreate replicaset/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf Created pod: e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v 20m Normal AddedInterface pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Add eth0 [10.134.1.124/23] from ovn-kubernetes 20m Normal Created pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Container created 20m Normal Killing pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Stopping container payload-pre-processing 20m Normal Killing pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Stopping container payload-processing 20m Normal Started pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Container started 20m Normal Pulled pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal ScalingReplicaSet deployment/e2e-shared-b-w4-674082-openshift-default Scaled up replica set e2e-shared-b-w4-674082-openshift-default-577d994559 from 3 to 4 20m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w4-674082-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 20m Normal SuccessfulCreate replicaset/e2e-shared-b-w4-674082-openshift-default-577d994559 Created pod: e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg 19m Normal Pulled pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Created pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Container created 19m Normal Started pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Container started 19m Normal AddedInterface pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Add eth0 [10.133.0.198/23] from ovn-kubernetes 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7 Error creating: pods "payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ec1091: serviceaccount "payload-processing-e2e-ipp-a-ec1091" not found 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-ec1091 Scaled up replica set payload-processing-e2e-ipp-a-ec1091-7bcb57d86d from 0 to 1 19m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d Error creating: pods "payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-ec1091: serviceaccount "payload-processing-e2e-ipp-a-ec1091" not found 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d Created pod: payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-ec1091 Scaled up replica set payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7 from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7 Created pod: payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 19m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Add eth0 [10.132.0.248/23] from ovn-kubernetes 19m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Startup probe failed: Get "http://10.133.0.198:15021/healthz/ready": dial tcp 10.133.0.198:15021: connect: connection refused 19m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Startup probe error: Get "http://10.133.0.198:15021/healthz/ready": dial tcp 10.133.0.198:15021: connect: connection refused... 19m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 19m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Startup probe error: Get "http://10.134.1.124:15021/healthz/ready": dial tcp 10.134.1.124:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Startup probe failed: Get "http://10.134.1.124:15021/healthz/ready": dial tcp 10.134.1.124:15021: connect: connection refused 19m Normal Started pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Container started 19m Normal Created pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Container created 19m Normal AddedInterface pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Add eth0 [10.134.1.127/23] from ovn-kubernetes 19m Normal Pulled pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 19m Normal Started pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Container started 19m Normal Created pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Container created 19m Warning Unhealthy pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal Killing pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-65l7v Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Stopping container istio-proxy 19m Normal Killing pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-xl7ml Stopping container istio-proxy 19m Normal Killing pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd Stopping container istio-proxy 19m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning ProbeError pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-pq25l Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning Unhealthy pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-fnbll Readiness probe failed: HTTP probe failed with statuscode: 503 19m Normal Killing pod/payload-pre-processing-e2e-ipp-b-4bc875-857bfffdcf-bfbs4 Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-ipp-b-4bc875-765d5b6fcb-zqkm4 Stopping container payload-processing 19m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w4-262c10-5cf855bbdc-w297h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal SuccessfulCreate replicaset/e2e-ipp-b-7a4600-openshift-default-bf9b5869c Created pod: e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf 19m Normal ScalingReplicaSet deployment/e2e-ipp-b-7a4600-openshift-default Scaled up replica set e2e-ipp-b-7a4600-openshift-default-bf9b5869c from 0 to 1 19m Warning Unhealthy pod/payload-processing-e2e-worker-w6-83e83e-544bdfd9d-nmzc5 Readiness probe failed: timeout: failed to connect service "10.134.0.54:50052" within 1s: context deadline exceeded 19m Normal AddedLabel gateway/e2e-ipp-b-7a4600 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-7a4600 19m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-83e83e-749bcb7575-bgm5z Readiness probe failed: timeout: failed to connect service "10.134.0.55:50052" within 1s: context deadline exceeded 19m Warning Unhealthy pod/payload-processing-e2e-shared-a-w4-262c10-bccc5986d-fwzfm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Normal AddedInterface pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Add eth0 [10.132.0.249/23] from ovn-kubernetes 19m Normal Killing pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Stopping container payload-processing 19m Normal ScalingReplicaSet deployment/e2e-ipp-a-2d21ab-openshift-default Scaled down replica set e2e-ipp-a-2d21ab-openshift-default-7d7974c69f from 9 to 6 19m Normal SuccessfulDelete replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Deleted pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz 19m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Stopping container istio-proxy 19m Normal SuccessfulRescale horizontalpodautoscaler/e2e-ipp-a-2d21ab-openshift-default New size: 6; reason: All metrics below target 19m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Stopping container istio-proxy 19m Normal SuccessfulDelete replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Deleted pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv 19m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Stopping container istio-proxy 19m Normal SuccessfulDelete replicaset/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f Deleted pod: e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c Error creating: pods "payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-7a4600: serviceaccount "payload-processing-e2e-ipp-b-7a4600" not found 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c Created pod: payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-7a4600 Scaled up replica set payload-processing-e2e-ipp-b-7a4600-686d87c79d from 0 to 1 19m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-7a4600-686d87c79d Error creating: pods "payload-processing-e2e-ipp-b-7a4600-686d87c79d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-7a4600: serviceaccount "payload-processing-e2e-ipp-b-7a4600" not found 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-7a4600 Scaled up replica set payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-7a4600-686d87c79d Created pod: payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f 19m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Add eth0 [10.134.1.130/23] from ovn-kubernetes 19m Normal Pulled pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal Created pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Container created 19m Normal Started pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Container started 19m Normal Created pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Container created 19m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bstzv Readiness probe failed: HTTP probe failed with statuscode: 503 19m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 19m Normal AddedInterface pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Add eth0 [10.134.1.131/23] from ovn-kubernetes 19m Normal Started pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Container started 19m Normal Created pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Container created 19m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Readiness probe error: HTTP probe failed with statuscode: 503... 19m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-ccl6c Readiness probe failed: HTTP probe failed with statuscode: 503 19m Normal Pulled pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Started pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Container started 19m Normal Killing pod/payload-processing-e2e-ipp-a-7226a8-68d55987b8-tvlr8 Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-ipp-a-7226a8-5b9dbcfdf5-hc5jn Stopping container payload-pre-processing 19m Warning Unhealthy pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Readiness probe failed: Get "http://10.134.1.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 19m Warning ProbeError pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-9l6pz Readiness probe error: Get "http://10.134.1.98:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 19m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs Stopping container istio-proxy 19m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w2-93bad9-openshift-default New size: 4; reason: All metrics below target 19m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-93bad9-openshift-default Scaled down replica set e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 from 6 to 4 19m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss Stopping container istio-proxy 19m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Deleted pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs 19m Normal SuccessfulDelete replicaset/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9 Deleted pod: e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss 19m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-4bc875-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-4bc875-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning ProbeError pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Startup probe error: Get "http://10.132.0.243:15021/healthz/ready": dial tcp 10.132.0.243:15021: connect: connection refused... 19m Warning Unhealthy pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Startup probe failed: Get "http://10.132.0.243:15021/healthz/ready": dial tcp 10.132.0.243:15021: connect: connection refused 19m Normal Pulled pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 19m Normal Created pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Container created 19m Normal Started pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Container started 19m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w4-674082-cbfbddc59-jr4r8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning Unhealthy pod/payload-processing-e2e-shared-b-w4-674082-6c4df5884d-vd2wt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-7226a8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-7226a8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning Unhealthy pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Startup probe failed: Get "http://10.134.1.115:15021/healthz/ready": dial tcp 10.134.1.115:15021: connect: connection refused 19m Warning ProbeError pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Startup probe error: Get "http://10.134.1.115:15021/healthz/ready": dial tcp 10.134.1.115:15021: connect: connection refused... 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-2d21ab-6b76d9f85f-f7mmz Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-a-2d21ab-5c4bd48fbb-xh5nq Stopping container payload-processing 18m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq Stopping container istio-proxy 18m Normal Killing pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-l25xz Stopping container istio-proxy 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-4bc875 Scaled up replica set payload-processing-e2e-ipp-b-4bc875-7c57d4dd46 from 0 to 1 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-4bc875 Scaled up replica set payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc from 0 to 1 18m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46 Error creating: pods "payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-4bc875: serviceaccount "payload-processing-e2e-ipp-b-4bc875" not found 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46 Created pod: payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc Created pod: payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc Error creating: pods "payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-4bc875: serviceaccount "payload-processing-e2e-ipp-b-4bc875" not found 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-4bc875-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Add eth0 [10.134.1.132/23] from ovn-kubernetes 18m Normal Pulled pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 18m Normal AddedInterface pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Add eth0 [10.134.1.133/23] from ovn-kubernetes 18m Normal Created pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Container created 18m Normal Started pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Container started 18m Normal Started pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Container started 18m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 18m Normal Created pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Container created 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-4bc875-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal Killing pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w4-674082-openshift-default-577d994559-pcb2w Stopping container istio-proxy 18m Normal Killing pod/e2e-shared-b-w4-674082-openshift-default-577d994559-89swj Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-644kz Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-b6dm5 Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bc4nt Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-bflb2 Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-dsrlr Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-zsppd Stopping container istio-proxy 18m Warning ProbeError pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/e2e-shared-b-w4-674082-openshift-default-577d994559-c6ghr Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-7226a8 Scaled up replica set payload-processing-e2e-ipp-a-7226a8-6b479cbc78 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9 Created pod: payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-a-7226a8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-7226a8 Scaled up replica set payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9 Error creating: pods "payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-7226a8: serviceaccount "payload-processing-e2e-ipp-a-7226a8" not found 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-7226a8-6b479cbc78 Created pod: payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 18m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-7226a8-6b479cbc78 Error creating: pods "payload-processing-e2e-ipp-a-7226a8-6b479cbc78-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-7226a8: serviceaccount "payload-processing-e2e-ipp-a-7226a8" not found 18m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-a-7226a8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Add eth0 [10.132.0.250/23] from ovn-kubernetes 18m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 18m Normal Pulled pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 18m Normal Created pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Container created 18m Normal Started pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Container started 18m Normal Created pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Container created 18m Normal Started pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Container started 18m Normal AddedInterface pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Add eth0 [10.134.1.137/23] from ovn-kubernetes 18m Normal Killing pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-b-b5b27e-65576cb866-7sj5p Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-ipp-b-b5b27e-bdbb4f6cf-fz6vk Stopping container payload-pre-processing 18m Normal SuccessfulCreate replicaset/e2e-ipp-b-a62fee-openshift-default-59bc6dff9 Created pod: e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 18m Normal ScalingReplicaSet deployment/e2e-ipp-b-a62fee-openshift-default Scaled up replica set e2e-ipp-b-a62fee-openshift-default-59bc6dff9 from 0 to 1 18m Normal AddedLabel gateway/e2e-ipp-b-a62fee Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-a62fee 18m Normal AddedInterface pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Add eth0 [10.132.0.251/23] from ovn-kubernetes 18m Normal Killing pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Stopping container istio-proxy 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7 Created pod: payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-a62fee Scaled up replica set payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7 from 0 to 1 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-a62fee Scaled up replica set payload-processing-e2e-ipp-b-a62fee-6b474cbc48 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-a62fee-6b474cbc48 Created pod: payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc 18m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-a62fee-6b474cbc48 Error creating: pods "payload-processing-e2e-ipp-b-a62fee-6b474cbc48-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a62fee: serviceaccount "payload-processing-e2e-ipp-b-a62fee" not found 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7 Error creating: pods "payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a62fee: serviceaccount "payload-processing-e2e-ipp-b-a62fee" not found 18m Normal Started pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Container started 18m Normal Created pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Container created 18m Normal AddedInterface pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Add eth0 [10.134.1.142/23] from ovn-kubernetes 18m Normal Created pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Container created 18m Normal Pulled pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 18m Normal Started pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Container started 18m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 18m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Add eth0 [10.134.1.143/23] from ovn-kubernetes 18m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-d55037-69f8c596c5-l5p8v Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/payload-processing-e2e-worker-w4-d55037-86f99856b-7gxz2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 18m Warning Unhealthy pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-ipp-b-b5b27e-openshift-default-7756fbd8f9-nqqq6 Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal Killing pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5 Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-ipp-b-7a4600-686d87c79d-mdf7f Stopping container payload-processing 18m Normal Killing pod/payload-pre-processing-e2e-ipp-b-7a4600-86cfdb6d8c-q2dnm Stopping container payload-pre-processing 18m Normal Pulled pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Started pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Container started 18m Warning ProbeError pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Readiness probe error: Get "http://10.132.0.249:15021/healthz/ready": dial tcp 10.132.0.249:15021: connect: connection refused... 18m Warning Unhealthy pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Readiness probe failed: Get "http://10.132.0.249:15021/healthz/ready": dial tcp 10.132.0.249:15021: connect: connection refused 18m Normal Created pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Container created 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-qjd49 Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-9krsz Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-sb67v Stopping container istio-proxy 18m Normal Killing pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Stopping container istio-proxy 18m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Readiness probe failed: HTTP probe failed with statuscode: 503 18m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-gpkg2 Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-99r7x Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning ProbeError pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Startup probe error: Get "http://10.132.0.249:15021/healthz/ready": dial tcp 10.132.0.249:15021: connect: connection refused... 17m Warning Unhealthy pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Startup probe failed: Get "http://10.132.0.249:15021/healthz/ready": dial tcp 10.132.0.249:15021: connect: connection refused 17m Warning ProbeError pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning Unhealthy pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-7a4600-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-7a4600-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning Unhealthy pod/payload-processing-e2e-shared-a-w0-ff4760-6d9d6754bb-6wlww Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-htpvd Stopping container istio-proxy 17m Normal Killing pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 Stopping container istio-proxy 17m Normal Killing pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb Stopping container istio-proxy 17m Normal Killing pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Stopping container istio-proxy 17m Normal Killing pod/payload-pre-processing-e2e-ipp-b-4bc875-7dd697f9bc-kk4f5 Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-ipp-b-4bc875-7c57d4dd46-j655g Stopping container payload-processing 17m Warning Unhealthy pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-85v9j Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7 Error creating: pods "payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-7a4600: serviceaccount "payload-processing-e2e-ipp-b-7a4600" not found 17m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7 Created pod: payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr 17m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-7a4600-868bff8dd4 Created pod: payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l 17m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-7a4600 Scaled up replica set payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7 from 0 to 1 17m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-7a4600-868bff8dd4 Error creating: pods "payload-processing-e2e-ipp-b-7a4600-868bff8dd4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-7a4600: serviceaccount "payload-processing-e2e-ipp-b-7a4600" not found 17m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-7a4600 Scaled up replica set payload-processing-e2e-ipp-b-7a4600-868bff8dd4 from 0 to 1 17m Normal Pulled pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Normal Started pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Container started 17m Normal Created pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Container created 17m Normal AddedInterface pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Add eth0 [10.132.0.254/23] from ovn-kubernetes 17m Normal Killing pod/e2e-ipp-b-4bc875-openshift-default-5798f46f5f-tslrg Stopping container istio-proxy 17m Normal Started pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Container started 17m Normal Created pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Container created 17m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 17m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Add eth0 [10.132.0.253/23] from ovn-kubernetes 17m Normal Killing pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Stopping container payload-processing 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-7a4600-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-7a4600-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Stopping container payload-pre-processing 17m Normal Killing pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Stopping container payload-processing 17m Normal Started pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Container started 17m Normal Pulled pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 17m Normal Created pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Container created 17m Normal Killing pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Stopping container istio-proxy 17m Normal SuccessfulDelete replicaset/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 Deleted pod: e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m 17m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w5-34722e-openshift-default New size: 3; reason: All metrics below target 17m Normal ScalingReplicaSet deployment/e2e-shared-a-w5-34722e-openshift-default Scaled down replica set e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95 from 4 to 3 17m Warning Unhealthy pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Readiness probe failed: HTTP probe failed with statuscode: 503 17m Warning ProbeError pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-2cn2m Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-a62fee-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-a62fee-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning Unhealthy pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Startup probe failed: Get "http://10.132.0.251:15021/healthz/ready": dial tcp 10.132.0.251:15021: connect: connection refused 17m Warning ProbeError pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Startup probe error: Get "http://10.132.0.251:15021/healthz/ready": dial tcp 10.132.0.251:15021: connect: connection refused... 17m Normal Killing pod/payload-processing-e2e-ipp-b-a62fee-6b474cbc48-kc9kc Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-ipp-b-a62fee-6b5f48d5d7-c97xb Stopping container payload-pre-processing 17m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w0-dfb231-58c4bdb8d7-gz6pl Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 17m Warning Unhealthy pod/payload-processing-e2e-shared-b-w0-dfb231-b5867f45d-kw9hd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-q2z6c Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd Stopping container istio-proxy 16m Warning Unhealthy pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-kn5sg Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning ProbeError pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-shared-a-w2-44bc69-openshift-default-74d8f6cf9-bf8jn Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Stopping container payload-processing 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-ipp-b-a62fee-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-ipp-b-a62fee-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Normal Killing pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Stopping container payload-processing 16m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-a62fee Scaled up replica set payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4 from 0 to 1 16m Normal Killing pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Stopping container payload-pre-processing 16m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587 Error creating: pods "payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a62fee: serviceaccount "payload-processing-e2e-ipp-b-a62fee" not found 16m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587 Created pod: payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp 16m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-a62fee Scaled up replica set payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587 from 0 to 1 16m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4 Created pod: payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 16m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4 Error creating: pods "payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-a62fee: serviceaccount "payload-processing-e2e-ipp-b-a62fee" not found 16m Normal Killing pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Stopping container payload-processing 16m Normal Created pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Container created 16m Warning Unhealthy pod/payload-processing-e2e-ipp-a-ec1091-7bcb57d86d-ffcr8 Readiness probe failed: dial tcp 10.134.1.127:9004: connect: connection refused 16m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Add eth0 [10.132.0.255/23] from ovn-kubernetes 16m Normal Created pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Container created 16m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 16m Normal Started pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Container started 16m Normal Started pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Container started 16m Normal Killing pod/payload-pre-processing-e2e-ipp-a-ec1091-556f9976d7-hngf8 Stopping container payload-pre-processing 16m Normal AddedInterface pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Add eth0 [10.132.1.0/23] from ovn-kubernetes 16m Normal Pulled pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine and can be accessed by the pod 16m Normal Killing pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Stopping container istio-proxy 16m Normal SuccessfulDelete replicaset/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc Deleted pod: e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx 16m Normal Killing pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Stopping container istio-proxy 16m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w5-0dba31-openshift-default New size: 2; reason: All metrics below target 16m Normal ScalingReplicaSet deployment/e2e-shared-b-w5-0dba31-openshift-default Scaled down replica set e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc from 3 to 2 16m Warning Unhealthy pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-ipp-a-ec1091-openshift-default-5b9f98669-8c2cq Readiness probe error: HTTP probe failed with statuscode: 503... 16m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw Stopping container istio-proxy 16m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default New size: 7; reason: All metrics below target 16m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled down replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 10 to 7 16m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh Stopping container istio-proxy 16m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Deleted pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sd8gw 16m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Deleted pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf 16m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l5nnf Stopping container istio-proxy 16m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Deleted pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-lwmlh 16m Warning ProbeError pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-cxbmx Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning Unhealthy pod/payload-processing-e2e-shared-b-w2-93bad9-547d6f96f8-4bspk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w2-93bad9-5556f888cf-s2fjk Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 16m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-ph5fz Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Stopping container istio-proxy 16m Normal Killing pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-58d7f Stopping container istio-proxy 16m Warning Unhealthy pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-gwt6n Readiness probe error: HTTP probe failed with statuscode: 503... 16m Normal Killing pod/payload-pre-processing-e2e-ipp-b-7a4600-5dfbbcffd7-bjzsr Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-ipp-a-7226a8-6b479cbc78-w48z9 Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-ipp-a-7226a8-59bd5659f9-b4t8s Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-ipp-b-7a4600-868bff8dd4-ddp7l Stopping container payload-processing 16m Normal Killing pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Stopping container payload-pre-processing 16m Normal Killing pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Stopping container istio-proxy 15m Normal Killing pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Stopping container istio-proxy 15m Warning Unhealthy pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-ipp-b-7a4600-openshift-default-bf9b5869c-cs7gf Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-ipp-a-7226a8-openshift-default-bf9bc5767-vmc5w Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Stopping container payload-pre-processing 15m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 Stopping container istio-proxy 15m Normal Killing pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-cnj5d Stopping container istio-proxy 15m Normal Killing pod/payload-processing-e2e-ipp-b-a62fee-5bddfdf8f4-fpsf9 Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-ipp-b-a62fee-5cc88cd587-55twp Stopping container payload-pre-processing 15m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w2-397179-7bcffdf8c-dwl72 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-processing-e2e-worker-w2-397179-6db8cdbc74-6sfx4 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-35a94e-openshift-default Scaled down replica set e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc from 8 to 5 15m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-a-w1-35a94e-openshift-default New size: 5; reason: All metrics below target 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Stopping container istio-proxy 15m Normal SuccessfulDelete replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Deleted pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz 15m Normal SuccessfulDelete replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Deleted pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb 15m Normal SuccessfulDelete replicaset/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc Deleted pod: e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-ghqqz Stopping container istio-proxy 15m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w5-34722e-5bdd47b4b7-w2mqz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-processing-e2e-shared-a-w5-34722e-54578dd88-bqd8q Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Normal Killing pod/e2e-ipp-b-a62fee-openshift-default-59bc6dff9-g95l6 Stopping container istio-proxy 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Readiness probe error: Get "http://10.133.0.164:15021/healthz/ready": dial tcp 10.133.0.164:15021: connect: connection refused... 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Readiness probe failed: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb Readiness probe error: Get "http://10.133.0.165:15021/healthz/ready": dial tcp 10.133.0.165:15021: connect: connection refused... 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh Readiness probe failed: Get "http://10.133.0.164:15021/healthz/ready": dial tcp 10.133.0.164:15021: connect: connection refused 15m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-m6jdn Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-sl78v Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-gg99v Stopping container istio-proxy 15m Warning Unhealthy pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw Stopping container istio-proxy 15m Normal Killing pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Stopping container payload-pre-processing 15m Normal Killing pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Stopping container payload-pre-processing 15m Normal Killing pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Stopping container payload-processing 15m Normal Killing pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-jr6jn Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-sh8fp Stopping container istio-proxy 15m Warning Unhealthy pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-dhttw Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Stopping container istio-proxy 15m Normal Killing pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Stopping container istio-proxy 15m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w5-0dba31-c6f69c747-5pczs Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning Unhealthy pod/payload-processing-e2e-shared-b-w5-0dba31-b779bf6cd-p7vkv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-l7768 Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-2crqh Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Readiness probe failed: HTTP probe failed with statuscode: 503 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-fwqnm Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning ProbeError pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Readiness probe error: HTTP probe failed with statuscode: 503... 15m Warning Unhealthy pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 Readiness probe failed: HTTP probe failed with statuscode: 503 14m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-9dc17a-openshift-default Scaled down replica set e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 from 7 to 6 14m Normal SuccessfulDelete replicaset/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45 Deleted pod: e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq Stopping container istio-proxy 14m Normal SuccessfulRescale horizontalpodautoscaler/e2e-shared-b-w1-9dc17a-openshift-default New size: 6; reason: All metrics below target 14m Normal Killing pod/payload-processing-e2e-worker-w5-0b21ba-8596dbbf4d-zflbp Stopping container payload-processing 14m Normal Killing pod/payload-pre-processing-e2e-worker-w5-0b21ba-7b5b644bb7-2qk8k Stopping container payload-pre-processing 14m Warning Unhealthy pod/payload-processing-e2e-shared-a-w1-35a94e-6996dd895b-2kzfd Readiness probe failed: timeout: failed to connect service "10.134.1.33:50052" within 1s: context deadline exceeded 14m Warning Unhealthy pod/payload-pre-processing-e2e-shared-a-w1-35a94e-79bd6d8767-6lrzz Readiness probe failed: timeout: failed to connect service "10.134.1.32:50052" within 1s: context deadline exceeded 14m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Stopping container payload-pre-processing 14m Normal Killing pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Stopping container payload-processing 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-ftnqd Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Stopping container istio-proxy 14m Normal Killing pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Stopping container istio-proxy 14m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-4s8v9 Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-l8tcr Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning ProbeError pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning Unhealthy pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-8xv8q Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning Unhealthy pod/payload-processing-e2e-shared-b-w1-9dc17a-c856945c8-4rp8h Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Warning Unhealthy pod/payload-pre-processing-e2e-shared-b-w1-9dc17a-5b7ccfdf79-frwcb Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal Killing pod/payload-processing-e2e-worker-w1-cff4ac-7b5bb7fcdb-rrdxr Stopping container payload-processing 14m Normal Killing pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Stopping container payload-pre-processing 14m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw Stopping container istio-proxy 14m Normal Killing pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-dx25f Stopping container istio-proxy 14m Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-cff4ac-56cff8d6f5-qm4jn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 14m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-bx4nb 14m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb Stopping container discovery 14m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 3 to 2 14m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: All metrics below target 14m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Stopping container istio-proxy 14m Normal Killing pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Stopping container istio-proxy 14m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Readiness probe failed: HTTP probe failed with statuscode: 503 14m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning ProbeError pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Readiness probe error: HTTP probe failed with statuscode: 503... 14m Warning Unhealthy pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-8pqbc Readiness probe failed: HTTP probe failed with statuscode: 503 13m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 3 to 6 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-68n2k 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Add eth0 [10.133.0.201/23] from ovn-kubernetes 13m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Container created 13m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Add eth0 [10.132.1.2/23] from ovn-kubernetes 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Add eth0 [10.134.1.166/23] from ovn-kubernetes 13m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Container created 13m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Container created 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Startup probe error: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused... 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Startup probe error: Get "http://10.134.1.166:15021/healthz/ready": dial tcp 10.134.1.166:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Startup probe failed: Get "http://10.134.1.166:15021/healthz/ready": dial tcp 10.134.1.166:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Startup probe error: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Startup probe failed: Get "http://10.133.0.201:15021/healthz/ready": dial tcp 10.133.0.201:15021: connect: connection refused 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Startup probe failed: Get "http://10.132.1.2:15021/healthz/ready": dial tcp 10.132.1.2:15021: connect: connection refused 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-wprbn 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-db25s 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6f77ddbcb7 Created pod: maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6f77ddbcb7 from 6 to 10 12m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Container created 12m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Container started 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Add eth0 [10.133.0.202/23] from ovn-kubernetes 12m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Container started 12m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Add eth0 [10.134.1.169/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Container created 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Add eth0 [10.132.1.4/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Container started 12m Normal Started pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Container started 12m Normal Created pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Add eth0 [10.132.1.5/23] from ovn-kubernetes 12m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Startup probe error: Get "http://10.133.0.202:15021/healthz/ready": dial tcp 10.133.0.202:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Startup probe error: Get "http://10.132.1.5:15021/healthz/ready": dial tcp 10.132.1.5:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Startup probe failed: Get "http://10.132.1.4:15021/healthz/ready": dial tcp 10.132.1.4:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Startup probe error: Get "http://10.132.1.4:15021/healthz/ready": dial tcp 10.132.1.4:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Startup probe failed: Get "http://10.132.1.5:15021/healthz/ready": dial tcp 10.132.1.5:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Startup probe failed: Get "http://10.134.1.169:15021/healthz/ready": dial tcp 10.134.1.169:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Startup probe error: Get "http://10.134.1.169:15021/healthz/ready": dial tcp 10.134.1.169:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Startup probe failed: Get "http://10.133.0.202:15021/healthz/ready": dial tcp 10.133.0.202:15021: connect: connection refused 12m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7ddd9cfdc8 Deleted pod: istiod-openshift-gateway-7ddd9cfdc8-842vl 12m Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 1; reason: All metrics below target 12m Normal Killing pod/istiod-openshift-gateway-7ddd9cfdc8-842vl Stopping container discovery 12m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7ddd9cfdc8 from 2 to 1 9m8s Normal AddedLabel gateway/e2e-worker-single-main-f97a49 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-f97a49 9m8s Normal AddedInterface pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Add eth0 [10.132.1.6/23] from ovn-kubernetes 9m8s Normal ScalingReplicaSet deployment/e2e-worker-single-main-f97a49-openshift-default Scaled up replica set e2e-worker-single-main-f97a49-openshift-default-6b74b856d9 from 0 to 1 9m8s Normal SuccessfulCreate replicaset/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9 Created pod: e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb 9m3s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-f97a49 Scaled up replica set payload-processing-e2e-worker-single-main-f97a49-76df79cb9b from 0 to 1 9m3s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-f97a49-76df79cb9b Created pod: payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj 9m3s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-f97a49 Scaled up replica set payload-pre-processing-e2e-worker-single-main-f97a49-858887fb79 from 0 to 1 9m3s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-f97a49-858887fb79 Error creating: pods "payload-pre-processing-e2e-worker-single-main-f97a49-858887fb79-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f97a49: serviceaccount "payload-processing-e2e-worker-single-main-f97a49" not found 9m3s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-f97a49-76df79cb9b Error creating: pods "payload-processing-e2e-worker-single-main-f97a49-76df79cb9b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-f97a49: serviceaccount "payload-processing-e2e-worker-single-main-f97a49" not found 9m2s Normal Created pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Container created 9m2s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Add eth0 [10.134.1.181/23] from ovn-kubernetes 9m2s Normal Pulled pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m2s Normal Started pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Container started 9m2s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Add eth0 [10.134.1.182/23] from ovn-kubernetes 9m2s Normal Created pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Container created 9m2s Normal Started pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Container started 9m2s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 9m2s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-f97a49-858887fb79 Created pod: payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx 8m50s Warning ProbeError pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Readiness probe error: Get "http://10.132.1.6:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 8m50s Warning Unhealthy pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Readiness probe failed: Get "http://10.132.1.6:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 8m49s Normal Pulled pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m49s Normal Created pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Container created 8m49s Normal Started pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Container started 8m43s Warning Unhealthy pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Startup probe failed: Get "http://10.132.1.6:15021/healthz/ready": dial tcp 10.132.1.6:15021: connect: connection refused 8m43s Warning ProbeError pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Startup probe error: Get "http://10.132.1.6:15021/healthz/ready": dial tcp 10.132.1.6:15021: connect: connection refused... 8m38s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-f97a49-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m38s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-f97a49-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m23s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-f97a49-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 8m23s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-f97a49-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 8m1s Normal Killing pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Stopping container payload-processing 8m1s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Stopping container payload-pre-processing 7m51s Normal Killing pod/e2e-worker-single-main-f97a49-openshift-default-6b74b856d9h72wb Stopping container istio-proxy 7m40s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-f97a49-85888p6jdx Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m40s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-f97a49-76df79cb965nbj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Readiness probe error: Get "http://10.132.1.4:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Readiness probe error: Get "http://10.134.1.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Readiness probe failed: Get "http://10.133.0.201:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Readiness probe error: Get "http://10.133.0.202:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb Readiness probe failed: Get "http://10.133.0.202:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Readiness probe error: Get "http://10.132.1.2:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Readiness probe error: Get "http://10.134.1.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-68n2k Readiness probe failed: Get "http://10.132.1.2:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x Readiness probe error: Get "http://10.133.0.201:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-wprbn Readiness probe failed: Get "http://10.132.1.4:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-jbc92 Readiness probe failed: Get "http://10.134.1.166:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-db25s Readiness probe failed: Get "http://10.134.1.169:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m5s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Readiness probe error: Get "http://10.132.1.5:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m5s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-dh5kx Readiness probe failed: Get "http://10.132.1.5:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m36s Normal AddedLabel gateway/e2e-worker-single-main-dcbb35 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-single-main-dcbb35 5m35s Normal AddedInterface pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Add eth0 [10.132.1.9/23] from ovn-kubernetes 5m35s Normal ScalingReplicaSet deployment/e2e-worker-single-main-dcbb35-openshift-default Scaled up replica set e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85 from 0 to 1 5m35s Normal SuccessfulCreate replicaset/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85 Created pod: e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 5m31s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-single-main-dcbb35 Scaled up replica set payload-processing-e2e-worker-single-main-dcbb35-5f87ddf94b from 0 to 1 5m30s Normal AddedInterface pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Add eth0 [10.134.1.190/23] from ovn-kubernetes 5m30s Normal Pulled pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m30s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-single-main-dcbb35-67944764bc Error creating: pods "payload-pre-processing-e2e-worker-single-main-dcbb35-67944764bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35: serviceaccount "payload-processing-e2e-worker-single-main-dcbb35" not found 5m30s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf94b Created pod: payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 5m30s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-single-main-dcbb35 Scaled up replica set payload-pre-processing-e2e-worker-single-main-dcbb35-67944764bc from 0 to 1 5m30s Warning FailedCreate replicaset/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf94b Error creating: pods "payload-processing-e2e-worker-single-main-dcbb35-5f87ddf94b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-single-main-dcbb35: serviceaccount "payload-processing-e2e-worker-single-main-dcbb35" not found 5m30s Normal Pulled pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m30s Normal AddedInterface pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Add eth0 [10.134.1.189/23] from ovn-kubernetes 5m30s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-single-main-dcbb35-67944764bc Created pod: payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn 5m30s Normal Created pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Container created 5m30s Normal Started pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Container started 5m29s Normal Created pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Container created 5m29s Normal Started pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Container started 5m18s Warning ProbeError pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Readiness probe error: Get "http://10.132.1.9:15021/healthz/ready": read tcp 10.132.0.2:51206->10.132.1.9:15021: read: connection reset by peer... 5m18s Warning Unhealthy pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Readiness probe failed: Get "http://10.132.1.9:15021/healthz/ready": read tcp 10.132.0.2:51206->10.132.1.9:15021: read: connection reset by peer 5m18s Normal Pulled pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m17s Normal Created pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Container created 5m17s Normal Started pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Container started 5m12s Warning ProbeError pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Startup probe error: Get "http://10.132.1.9:15021/healthz/ready": dial tcp 10.132.1.9:15021: connect: connection refused... 5m12s Warning Unhealthy pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Startup probe failed: Get "http://10.132.1.9:15021/healthz/ready": dial tcp 10.132.1.9:15021: connect: connection refused 5m5s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-dcbb35-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m5s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-dcbb35-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m35s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-single-main-dcbb35-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m35s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-single-main-dcbb35-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m34s Normal Killing pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Stopping container payload-processing 4m34s Normal Killing pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Stopping container payload-pre-processing 4m26s Warning Unhealthy pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Readiness probe failed: HTTP probe failed with statuscode: 503 4m26s Normal Killing pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Stopping container istio-proxy 4m26s Warning ProbeError pod/e2e-worker-single-main-dcbb35-openshift-default-5b5fb4d85-698w2 Readiness probe error: HTTP probe failed with statuscode: 503... 4m18s Warning Unhealthy pod/payload-pre-processing-e2e-worker-single-main-dcbb35-67944gqjpn Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m18s Warning Unhealthy pod/payload-processing-e2e-worker-single-main-dcbb35-5f87ddf9448655 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 3m30s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Readiness probe failed: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m30s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-52stt Readiness probe error: Get "http://10.134.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m13s Warning Unhealthy pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Readiness probe failed: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m13s Warning ProbeError pod/maas-default-gateway-openshift-default-6f77ddbcb7-4sxcs Readiness probe error: Get "http://10.132.0.70:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...