=== events in openshift-ingress (2026-09-19T15:55:24Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 18m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf to ip-10-0-141-0.ec2.internal 14m Normal Scheduled pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 to ip-10-0-141-0.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 to ip-10-0-141-0.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn to ip-10-0-136-237.ec2.internal 18m Normal Scheduled pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Successfully assigned openshift-ingress/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 to ip-10-0-132-152.ec2.internal 25m Normal Scheduled pod/payload-pre-processing-76f657cdc4-797pq Successfully assigned openshift-ingress/payload-pre-processing-76f657cdc4-797pq to ip-10-0-136-237.ec2.internal 20m Normal Scheduled pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Successfully assigned openshift-ingress/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft to ip-10-0-136-237.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w to ip-10-0-141-0.ec2.internal 23m Normal Scheduled pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Successfully assigned openshift-ingress/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 to ip-10-0-136-237.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx to ip-10-0-141-0.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw to ip-10-0-141-0.ec2.internal 14m Normal Scheduled pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 to ip-10-0-141-0.ec2.internal 13m Normal Scheduled pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Successfully assigned openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb to ip-10-0-141-0.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Successfully assigned openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q to ip-10-0-141-0.ec2.internal 25m Normal Scheduled pod/payload-processing-85fd65c94c-d9qjb Successfully assigned openshift-ingress/payload-processing-85fd65c94c-d9qjb to ip-10-0-136-237.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Successfully assigned openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Successfully assigned openshift-ingress/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv to ip-10-0-132-152.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz to ip-10-0-136-237.ec2.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw to ip-10-0-136-237.ec2.internal 33m Normal Scheduled pod/kube-auth-proxy-dc4565f8d-kgbgb Successfully assigned openshift-ingress/kube-auth-proxy-dc4565f8d-kgbgb to ip-10-0-136-237.ec2.internal 20m Normal Scheduled pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 to ip-10-0-136-237.ec2.internal 24m Normal Scheduled pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Successfully assigned openshift-ingress/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr to ip-10-0-132-152.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj to ip-10-0-141-0.ec2.internal 9m24s Normal Scheduled pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Successfully assigned openshift-ingress/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj to ip-10-0-132-152.ec2.internal 29m Normal Scheduled pod/payload-processing-5b54b7fbb7-zzvr5 Successfully assigned openshift-ingress/payload-processing-5b54b7fbb7-zzvr5 to ip-10-0-141-0.ec2.internal 25m Normal Scheduled pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Successfully assigned openshift-ingress/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 to ip-10-0-132-152.ec2.internal 33m Normal Scheduled pod/kube-auth-proxy-dc4565f8d-dnjjq Successfully assigned openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq to ip-10-0-141-0.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm to ip-10-0-141-0.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t to ip-10-0-136-237.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd to ip-10-0-136-237.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp to ip-10-0-141-0.ec2.internal 9m17s Normal Scheduled pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n to ip-10-0-141-0.ec2.internal 29m Normal Scheduled pod/payload-pre-processing-59fbc47bf8-wwcqx Successfully assigned openshift-ingress/payload-pre-processing-59fbc47bf8-wwcqx to ip-10-0-136-237.ec2.internal 14m Normal Scheduled pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Successfully assigned openshift-ingress/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh to ip-10-0-132-152.ec2.internal 33m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs to ip-10-0-132-152.ec2.internal 20m Normal Scheduled pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Successfully assigned openshift-ingress/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n to ip-10-0-136-237.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Successfully assigned openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz to ip-10-0-141-0.ec2.internal 11m Normal Scheduled pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Successfully assigned openshift-ingress/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 to ip-10-0-132-152.ec2.internal 19m Normal Scheduled pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Successfully assigned openshift-ingress/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb to ip-10-0-136-237.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx to ip-10-0-141-0.ec2.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw to ip-10-0-141-0.ec2.internal 13m Normal Scheduled pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Successfully assigned openshift-ingress/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r to ip-10-0-132-152.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Successfully assigned openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk to ip-10-0-141-0.ec2.internal 11m Normal Scheduled pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Successfully assigned openshift-ingress/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c to ip-10-0-132-152.ec2.internal 31m Normal Scheduled pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xx67 to ip-10-0-132-152.ec2.internal 19m Normal Scheduled pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Successfully assigned openshift-ingress/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf to ip-10-0-132-152.ec2.internal 39m Normal Scheduled pod/router-default-76d56ddb5b-flrvn Successfully assigned openshift-ingress/router-default-76d56ddb5b-flrvn to ip-10-0-136-237.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Successfully assigned openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Successfully assigned openshift-ingress/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q to ip-10-0-132-152.ec2.internal 9m17s Normal Scheduled pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Successfully assigned openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj to ip-10-0-141-0.ec2.internal 10m Normal Scheduled pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 to ip-10-0-141-0.ec2.internal 22m Normal Scheduled pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Successfully assigned openshift-ingress/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 to ip-10-0-136-237.ec2.internal 11m Normal Scheduled pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv to ip-10-0-141-0.ec2.internal 11m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t to ip-10-0-141-0.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d to ip-10-0-141-0.ec2.internal 19m Normal Scheduled pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Successfully assigned openshift-ingress/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h to ip-10-0-136-237.ec2.internal 25m Normal Scheduled pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Successfully assigned openshift-ingress/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf to ip-10-0-132-152.ec2.internal 19m Normal Scheduled pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Successfully assigned openshift-ingress/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 to ip-10-0-132-152.ec2.internal 33m Normal Scheduled pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-v6k5d to ip-10-0-136-237.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp to ip-10-0-141-0.ec2.internal 33m Normal Scheduled pod/istiod-openshift-gateway-5677ccb49b-pcbnn Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcbnn to ip-10-0-136-237.ec2.internal 25m Normal Scheduled pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Successfully assigned openshift-ingress/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q to ip-10-0-132-152.ec2.internal 49m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-76d56ddb5b from 0 to 1 49m Warning FailedCreate replicaset/router-default-76d56ddb5b Error creating: pods "router-default-76d56ddb5b-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 49m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 49m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 49m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 43m Warning FailedCreate replicaset/router-default-76d56ddb5b Error creating: pods "router-default-76d56ddb5b-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 43m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 39m Normal SuccessfulCreate replicaset/router-default-76d56ddb5b Created pod: router-default-76d56ddb5b-flrvn 39m Warning FailedMount pod/router-default-76d56ddb5b-flrvn MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 38m Warning FailedMount pod/router-default-76d56ddb5b-flrvn MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 38m Normal Created pod/router-default-76d56ddb5b-flrvn Created container: router 38m Normal Pulled pod/router-default-76d56ddb5b-flrvn Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f" already present on machine 38m Normal AddedInterface pod/router-default-76d56ddb5b-flrvn Add eth0 [10.132.0.15/23] from ovn-kubernetes 38m Normal Started pod/router-default-76d56ddb5b-flrvn Started container router 33m Normal SuccessfulCreate replicaset/kube-auth-proxy-dc4565f8d Created pod: kube-auth-proxy-dc4565f8d-kgbgb 33m Normal SuccessfulCreate replicaset/kube-auth-proxy-dc4565f8d Created pod: kube-auth-proxy-dc4565f8d-dnjjq 33m Warning FailedCreate replicaset/kube-auth-proxy-dc4565f8d Error creating: pods "kube-auth-proxy-dc4565f8d-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found 33m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-dc4565f8d from 0 to 2 33m Normal Pulling pod/kube-auth-proxy-dc4565f8d-kgbgb Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 33m Normal AddedInterface pod/kube-auth-proxy-dc4565f8d-dnjjq Add eth0 [10.134.0.19/23] from ovn-kubernetes 33m Normal Pulling pod/kube-auth-proxy-dc4565f8d-dnjjq Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 33m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7b7995d5cd Created pod: istiod-openshift-gateway-7b7995d5cd-v6k5d 33m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1 33m Normal AddedInterface pod/kube-auth-proxy-dc4565f8d-kgbgb Add eth0 [10.132.0.28/23] from ovn-kubernetes 33m Normal AddedInterface pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Add eth0 [10.132.0.29/23] from ovn-kubernetes 33m Normal Pulling pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" 33m Normal Pulled pod/kube-auth-proxy-dc4565f8d-dnjjq Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.663s (2.663s including waiting). Image size: 265120132 bytes. 33m Normal Started pod/kube-auth-proxy-dc4565f8d-dnjjq Started container kube-auth-proxy 33m Normal Pulled pod/kube-auth-proxy-dc4565f8d-kgbgb Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.765s (2.765s including waiting). Image size: 265120132 bytes. 33m Normal Created pod/kube-auth-proxy-dc4565f8d-kgbgb Created container: kube-auth-proxy 33m Normal Created pod/kube-auth-proxy-dc4565f8d-dnjjq Created container: kube-auth-proxy 33m Normal Started pod/kube-auth-proxy-dc4565f8d-kgbgb Started container kube-auth-proxy 33m Normal Started pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Started container discovery 33m Normal Pulled pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 3.276s (3.276s including waiting). Image size: 181126104 bytes. 33m Normal Created pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Created container: discovery 33m Warning Unhealthy pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Readiness probe failed: HTTP probe failed with statuscode: 503 33m Warning ProbeError pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Readiness probe error: HTTP probe failed with statuscode: 503... 33m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-d66cd8c7c Created pod: data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs 33m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-d66cd8c7c from 0 to 1 33m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Add eth0 [10.133.0.29/23] from ovn-kubernetes 33m Normal Pulling pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 33m Normal Pulled pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 5.263s (5.263s including waiting). Image size: 181318953 bytes. 33m Normal Started pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Started container istio-proxy 33m Normal Created pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-x85zs Created container: istio-proxy 33m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 33m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-5677ccb49b Created pod: istiod-openshift-gateway-5677ccb49b-pcbnn 33m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1 33m Normal Started pod/istiod-openshift-gateway-5677ccb49b-pcbnn Started container discovery 33m Normal Created pod/istiod-openshift-gateway-5677ccb49b-pcbnn Created container: discovery 33m Normal Pulled pod/istiod-openshift-gateway-5677ccb49b-pcbnn Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" already present on machine 33m Normal AddedInterface pod/istiod-openshift-gateway-5677ccb49b-pcbnn Add eth0 [10.132.0.30/23] from ovn-kubernetes 33m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0 33m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7b7995d5cd Deleted pod: istiod-openshift-gateway-7b7995d5cd-v6k5d 33m Normal Killing pod/istiod-openshift-gateway-7b7995d5cd-v6k5d Stopping container discovery 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 32m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 32m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-78bb78ff78 from 0 to 1 31m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 31m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-78bb78ff78 Created pod: maas-default-gateway-openshift-default-78bb78ff78-4xx67 31m Normal AddedInterface pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Add eth0 [10.133.0.40/23] from ovn-kubernetes 29m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-59fbc47bf8 from 0 to 1 29m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5b54b7fbb7 from 0 to 1 29m Warning FailedCreate replicaset/payload-processing-5b54b7fbb7 Error creating: pods "payload-processing-5b54b7fbb7-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 29m Normal SuccessfulCreate replicaset/payload-pre-processing-59fbc47bf8 Created pod: payload-pre-processing-59fbc47bf8-wwcqx 29m Normal AddedInterface pod/payload-pre-processing-59fbc47bf8-wwcqx Add eth0 [10.132.0.45/23] from ovn-kubernetes 29m Normal SuccessfulCreate replicaset/payload-processing-5b54b7fbb7 Created pod: payload-processing-5b54b7fbb7-zzvr5 29m Normal Pulling pod/payload-processing-5b54b7fbb7-zzvr5 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 29m Normal AddedInterface pod/payload-processing-5b54b7fbb7-zzvr5 Add eth0 [10.134.0.27/23] from ovn-kubernetes 29m Normal Pulling pod/payload-pre-processing-59fbc47bf8-wwcqx Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 29m Warning FailedCreate replicaset/payload-pre-processing-59fbc47bf8 Error creating: pods "payload-pre-processing-59fbc47bf8-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 29m Normal Created pod/payload-processing-5b54b7fbb7-zzvr5 Created container: payload-processing 29m Normal Started pod/payload-processing-5b54b7fbb7-zzvr5 Started container payload-processing 29m Normal Pulled pod/payload-processing-5b54b7fbb7-zzvr5 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.21s (1.21s including waiting). Image size: 166453408 bytes. 29m Normal Pulled pod/payload-pre-processing-59fbc47bf8-wwcqx Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.605s (1.605s including waiting). Image size: 166453408 bytes. 29m Normal Created pod/payload-pre-processing-59fbc47bf8-wwcqx Created container: payload-pre-processing 29m Normal Started pod/payload-pre-processing-59fbc47bf8-wwcqx Started container payload-pre-processing 29m Normal Pulled pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 29m Normal Created pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Created container: istio-proxy 29m Normal Started pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Started container istio-proxy 29m Warning ProbeError pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 29m Warning Unhealthy pod/maas-default-gateway-openshift-default-78bb78ff78-4xx67 Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 25m Normal Killing pod/payload-processing-5b54b7fbb7-zzvr5 Stopping container payload-processing 25m Normal Killing pod/payload-pre-processing-59fbc47bf8-wwcqx Stopping container payload-pre-processing 25m Normal SuccessfulCreate replicaset/payload-pre-processing-76f657cdc4 Created pod: payload-pre-processing-76f657cdc4-797pq 25m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-85fd65c94c from 0 to 1 25m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-76f657cdc4 from 0 to 1 25m Normal SuccessfulCreate replicaset/payload-processing-85fd65c94c Created pod: payload-processing-85fd65c94c-d9qjb 25m Normal AddedInterface pod/payload-pre-processing-76f657cdc4-797pq Add eth0 [10.132.0.51/23] from ovn-kubernetes 25m Normal Pulling pod/payload-pre-processing-76f657cdc4-797pq Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 25m Normal AddedInterface pod/payload-processing-85fd65c94c-d9qjb Add eth0 [10.132.0.50/23] from ovn-kubernetes 25m Normal Pulling pod/payload-processing-85fd65c94c-d9qjb Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 25m Normal Created pod/payload-pre-processing-76f657cdc4-797pq Created container: payload-pre-processing 25m Normal Started pod/payload-pre-processing-76f657cdc4-797pq Started container payload-pre-processing 25m Normal Pulled pod/payload-processing-85fd65c94c-d9qjb Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.094s (1.094s including waiting). Image size: 141049838 bytes. 25m Normal Created pod/payload-processing-85fd65c94c-d9qjb Created container: payload-processing 25m Normal Started pod/payload-processing-85fd65c94c-d9qjb Started container payload-processing 25m Normal Pulled pod/payload-pre-processing-76f657cdc4-797pq Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.049s (1.049s including waiting). Image size: 141049838 bytes. 25m Normal ScalingReplicaSet deployment/e2e-worker-w3-33737b-openshift-default Scaled up replica set e2e-worker-w3-33737b-openshift-default-58b799f4b4 from 0 to 1 25m Normal AddedLabel gateway/e2e-worker-w3-33737b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-33737b 25m Normal SuccessfulCreate replicaset/e2e-worker-w3-33737b-openshift-default-58b799f4b4 Created pod: e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf 25m Normal ScalingReplicaSet deployment/e2e-inf-a-21d2a4-openshift-default Scaled up replica set e2e-inf-a-21d2a4-openshift-default-6447f5dd4b from 0 to 1 25m Normal AddedLabel gateway/e2e-inf-a-21d2a4 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-21d2a4 25m Normal AddedLabel gateway/e2e-worker-w0-15949d Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-15949d 25m Normal AddedInterface pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Add eth0 [10.133.0.45/23] from ovn-kubernetes 25m Normal SuccessfulCreate replicaset/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76 Created pod: e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 25m Normal ScalingReplicaSet deployment/e2e-worker-w0-15949d-openshift-default Scaled up replica set e2e-worker-w0-15949d-openshift-default-7b5f7fcf76 from 0 to 1 25m Normal SuccessfulCreate replicaset/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b Created pod: e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q 25m Normal AddedInterface pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Add eth0 [10.133.0.46/23] from ovn-kubernetes 25m Normal AddedInterface pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Add eth0 [10.133.0.47/23] from ovn-kubernetes 25m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-21d2a4 Scaled up replica set payload-processing-e2e-inf-a-21d2a4-58d9f9c75c from 0 to 1 25m Normal Pulled pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 25m Normal Created pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Created container: payload-pre-processing 25m Normal Started pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Started container payload-processing 25m Normal Created pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Created container: payload-processing 25m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c Created pod: payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd 25m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Add eth0 [10.134.0.33/23] from ovn-kubernetes 25m Normal Pulled pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 25m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58 Created pod: payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm 25m Normal AddedInterface pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Add eth0 [10.132.0.53/23] from ovn-kubernetes 25m Normal Started pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Started container payload-pre-processing 25m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58 Error creating: pods "payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-21d2a4: serviceaccount "payload-processing-e2e-inf-a-21d2a4" not found 25m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-21d2a4 Scaled up replica set payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58 from 0 to 1 25m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c Error creating: pods "payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-21d2a4: serviceaccount "payload-processing-e2e-inf-a-21d2a4" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-15949d Scaled up replica set payload-processing-e2e-worker-w0-15949d-5694879547 from 0 to 1 24m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-15949d-5694879547 Error creating: pods "payload-processing-e2e-worker-w0-15949d-5694879547-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-15949d: serviceaccount "payload-processing-e2e-worker-w0-15949d" not found 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-15949d Scaled up replica set payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd Created pod: payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd Error creating: pods "payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-15949d: serviceaccount "payload-processing-e2e-worker-w0-15949d" not found 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-15949d-5694879547 Created pod: payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz 24m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Add eth0 [10.134.0.34/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Add eth0 [10.134.0.35/23] from ovn-kubernetes 24m Normal Created pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Created container: payload-processing 24m Normal Started pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Started container payload-pre-processing 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-33737b Scaled up replica set payload-pre-processing-e2e-worker-w3-33737b-94d6b7658 from 0 to 1 24m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Add eth0 [10.134.0.36/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658 Created pod: payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658 Error creating: pods "payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-33737b: serviceaccount "payload-processing-e2e-worker-w3-33737b" not found 24m Normal Started pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Started container payload-pre-processing 24m Normal Created pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Created container: payload-pre-processing 24m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Created container: payload-pre-processing 24m Normal Started pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Started container payload-processing 24m Normal Pulled pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Created container: payload-processing 24m Normal Pulled pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-33737b-76cc56d95f Error creating: pods "payload-processing-e2e-worker-w3-33737b-76cc56d95f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-33737b: serviceaccount "payload-processing-e2e-worker-w3-33737b" not found 24m Normal Started pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Started container payload-processing 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-33737b Scaled up replica set payload-processing-e2e-worker-w3-33737b-76cc56d95f from 0 to 1 24m Normal AddedInterface pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Add eth0 [10.134.0.37/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-33737b-76cc56d95f Created pod: payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx 24m Normal AddedLabel gateway/e2e-ait-e86b6784 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-e86b6784 24m Normal ScalingReplicaSet deployment/e2e-ait-e86b6784-openshift-default Scaled up replica set e2e-ait-e86b6784-openshift-default-fdcf56bd8 from 0 to 1 24m Normal SuccessfulCreate replicaset/e2e-ait-e86b6784-openshift-default-fdcf56bd8 Created pod: e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr 24m Normal AddedInterface pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Add eth0 [10.133.0.49/23] from ovn-kubernetes 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-e86b6784 Scaled up replica set payload-pre-processing-e2e-ait-e86b6784-54d7dc56db from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-e86b6784 Scaled up replica set payload-processing-e2e-ait-e86b6784-68f95d9f95 from 0 to 1 24m Normal AddedInterface pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Add eth0 [10.134.0.38/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Add eth0 [10.134.0.39/23] from ovn-kubernetes 24m Normal Pulled pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Created container: payload-pre-processing 24m Normal Pulled pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Created container: payload-processing 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-e86b6784-68f95d9f95 Created pod: payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk 24m Normal Started pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Started container payload-pre-processing 24m Normal Started pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Started container payload-processing 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db Created pod: payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx 24m Warning FailedCreate replicaset/payload-processing-e2e-ait-e86b6784-68f95d9f95 Error creating: pods "payload-processing-e2e-ait-e86b6784-68f95d9f95-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-e86b6784: serviceaccount "payload-processing-e2e-ait-e86b6784" not found 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db Error creating: pods "payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-e86b6784: serviceaccount "payload-processing-e2e-ait-e86b6784" not found 24m Normal SuccessfulCreate replicaset/e2e-inf-b-e93f57-openshift-default-744dd4d998 Created pod: e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q 24m Normal AddedLabel gateway/e2e-inf-b-e93f57 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-e93f57 24m Warning Unhealthy pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Readiness probe failed: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal AddedInterface pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Add eth0 [10.133.0.50/23] from ovn-kubernetes 24m Normal ScalingReplicaSet deployment/e2e-inf-b-e93f57-openshift-default Scaled up replica set e2e-inf-b-e93f57-openshift-default-744dd4d998 from 0 to 1 24m Warning ProbeError pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Readiness probe error: Get "http://10.133.0.45:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Warning ProbeError pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Readiness probe error: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Warning Unhealthy pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Readiness probe failed: Get "http://10.133.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal Created pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Created container: istio-proxy 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d Error creating: pods "payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e93f57: serviceaccount "payload-processing-e2e-inf-b-e93f57" not found 24m Normal Started pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Started container istio-proxy 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-e93f57 Scaled up replica set payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-e93f57 Scaled up replica set payload-processing-e2e-inf-b-e93f57-5fff958f57 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-e93f57-5fff958f57 Created pod: payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t 24m Normal Started pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Started container istio-proxy 24m Normal Created pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Created container: istio-proxy 24m Normal Pulled pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 24m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-e93f57-5fff958f57 Error creating: pods "payload-processing-e2e-inf-b-e93f57-5fff958f57-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-e93f57: serviceaccount "payload-processing-e2e-inf-b-e93f57" not found 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d Created pod: payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm 24m Normal Started pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Started container istio-proxy 24m Normal Created pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Created container: istio-proxy 24m Normal Pulled pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 24m Normal Pulled pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 24m Normal Pulled pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Started pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Started container payload-pre-processing 24m Normal Created pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Created container: payload-pre-processing 24m Normal AddedInterface pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Add eth0 [10.134.0.41/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Add eth0 [10.134.0.40/23] from ovn-kubernetes 24m Normal Started pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Started container payload-processing 24m Normal Pulled pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Created container: payload-processing 24m Warning Unhealthy pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused 24m Warning ProbeError pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused... 24m Warning ProbeError pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Startup probe failed: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused 24m Warning Unhealthy pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused 24m Warning ProbeError pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Startup probe error: Get "http://10.133.0.45:15021/healthz/ready": dial tcp 10.133.0.45:15021: connect: connection refused... 24m Normal Pulled pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 24m Normal Created pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Created container: istio-proxy 24m Normal Started pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Started container istio-proxy 24m Warning ProbeError pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 24m Warning Unhealthy pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 24m Normal Started pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Started container istio-proxy 24m Normal Created pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Created container: istio-proxy 24m Normal Pulled pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 24m Warning Unhealthy pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 24m Warning ProbeError pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 24m Normal Killing pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Stopping container istio-proxy 24m Normal Killing pod/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk Stopping container payload-processing 24m Normal Killing pod/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx Stopping container payload-pre-processing 24m Normal SuccessfulCreate replicaset/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb Created pod: e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv 24m Normal ScalingReplicaSet deployment/e2e-migrate-a95f496f-gw-openshift-default Scaled up replica set e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb from 0 to 1 24m Normal AddedLabel gateway/e2e-migrate-a95f496f-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-a95f496f-gw 24m Normal AddedInterface pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Add eth0 [10.133.0.52/23] from ovn-kubernetes 24m Warning Unhealthy pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Readiness probe failed: HTTP probe failed with statuscode: 503 24m Warning ProbeError pod/e2e-ait-e86b6784-openshift-default-fdcf56bd8-qznmr Readiness probe error: HTTP probe failed with statuscode: 503... 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-a95f496f Scaled up replica set payload-pre-processing-e2e-migrate-a95f496f-6b499454fd from 0 to 1 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd Error creating: pods "payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-a95f496f: serviceaccount "payload-processing-e2e-migrate-a95f496f" not found 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-a95f496f Scaled up replica set payload-processing-e2e-migrate-a95f496f-64ccbf6798 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-a95f496f-64ccbf6798 Created pod: payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh 24m Warning FailedCreate replicaset/payload-processing-e2e-migrate-a95f496f-64ccbf6798 Error creating: pods "payload-processing-e2e-migrate-a95f496f-64ccbf6798-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-a95f496f: serviceaccount "payload-processing-e2e-migrate-a95f496f" not found 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd Created pod: payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw 24m Normal Pulled pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Started pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Started container payload-pre-processing 24m Normal Started pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Started container payload-processing 24m Normal Created pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Created container: payload-pre-processing 24m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Add eth0 [10.134.0.44/23] from ovn-kubernetes 24m Normal Created pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Created container: payload-processing 24m Normal AddedInterface pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Add eth0 [10.134.0.45/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Warning ProbeError pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Readiness probe error: Get "http://10.133.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 23m Warning Unhealthy pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Readiness probe failed: Get "http://10.133.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 23m Normal Pulled pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Created pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Created container: istio-proxy 23m Normal Started pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Started container istio-proxy 23m Warning ProbeError pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused 23m Normal Killing pod/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh Stopping container payload-processing 23m Normal Killing pod/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw Stopping container payload-pre-processing 23m Normal Killing pod/e2e-migrate-a95f496f-gw-openshift-default-856fbb8fcb-9rccv Stopping container istio-proxy 23m Normal AddedLabel gateway/e2e-ait-63d8f735 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-63d8f735 23m Normal ScalingReplicaSet deployment/e2e-ait-63d8f735-openshift-default Scaled up replica set e2e-ait-63d8f735-openshift-default-554b98bc84 from 0 to 1 23m Normal SuccessfulCreate replicaset/e2e-ait-63d8f735-openshift-default-554b98bc84 Created pod: e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 23m Normal AddedInterface pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Add eth0 [10.132.0.59/23] from ovn-kubernetes 23m Normal Pulling pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 23m Normal Pulled pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.902s (2.902s including waiting). Image size: 181318953 bytes. 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-63d8f735-c78df56c8 Error creating: pods "payload-pre-processing-e2e-ait-63d8f735-c78df56c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-63d8f735: serviceaccount "payload-processing-e2e-ait-63d8f735" not found 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-63d8f735-c78df56c8 Created pod: payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-63d8f735 Scaled up replica set payload-pre-processing-e2e-ait-63d8f735-c78df56c8 from 0 to 1 23m Warning FailedCreate replicaset/payload-processing-e2e-ait-63d8f735-7775688b7f Error creating: pods "payload-processing-e2e-ait-63d8f735-7775688b7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-63d8f735: serviceaccount "payload-processing-e2e-ait-63d8f735" not found 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-63d8f735 Scaled up replica set payload-processing-e2e-ait-63d8f735-7775688b7f from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-63d8f735-7775688b7f Created pod: payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl 23m Normal Pulled pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Normal AddedInterface pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Add eth0 [10.134.0.47/23] from ovn-kubernetes 23m Normal Started pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Started container payload-processing 23m Normal Created pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Created container: payload-processing 23m Normal Pulled pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Normal AddedInterface pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Add eth0 [10.134.0.48/23] from ovn-kubernetes 23m Normal Started pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Started container payload-pre-processing 23m Normal Created pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Created container: payload-pre-processing 22m Normal Pulled pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 22m Normal Started pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Started container istio-proxy 22m Normal Created pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Created container: istio-proxy 22m Warning ProbeError pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Startup probe error: Get "http://10.132.0.59:15021/healthz/ready": dial tcp 10.132.0.59:15021: connect: connection refused... 22m Warning Unhealthy pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Startup probe failed: Get "http://10.132.0.59:15021/healthz/ready": dial tcp 10.132.0.59:15021: connect: connection refused 22m Normal Killing pod/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl Stopping container payload-processing 22m Normal Killing pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Stopping container istio-proxy 22m Normal AddedLabel gateway/e2e-derive-24d51017 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-24d51017 22m Normal AddedInterface pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Add eth0 [10.132.0.61/23] from ovn-kubernetes 22m Normal SuccessfulCreate replicaset/e2e-derive-24d51017-openshift-default-57c46d584d Created pod: e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 22m Normal ScalingReplicaSet deployment/e2e-derive-24d51017-openshift-default Scaled up replica set e2e-derive-24d51017-openshift-default-57c46d584d from 0 to 1 22m Warning ProbeError pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning Unhealthy pod/e2e-ait-63d8f735-openshift-default-554b98bc84-mvxk8 Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-24d51017-8596956b98 Created pod: payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x 22m Warning FailedCreate replicaset/payload-processing-e2e-derive-24d51017-6b5b6d67df Error creating: pods "payload-processing-e2e-derive-24d51017-6b5b6d67df-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-24d51017: serviceaccount "payload-processing-e2e-derive-24d51017" not found 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-24d51017 Scaled up replica set payload-processing-e2e-derive-24d51017-6b5b6d67df from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-24d51017-8596956b98 Error creating: pods "payload-pre-processing-e2e-derive-24d51017-8596956b98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-24d51017: serviceaccount "payload-processing-e2e-derive-24d51017" not found 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-24d51017-6b5b6d67df Created pod: payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-24d51017 Scaled up replica set payload-pre-processing-e2e-derive-24d51017-8596956b98 from 0 to 1 22m Normal AddedInterface pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Add eth0 [10.134.0.51/23] from ovn-kubernetes 22m Normal Pulled pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal AddedInterface pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Add eth0 [10.134.0.52/23] from ovn-kubernetes 22m Normal Pulled pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal Created pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Created container: payload-processing 22m Normal Started pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Started container payload-pre-processing 22m Normal Started pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Started container payload-processing 22m Normal Created pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Created container: payload-pre-processing 22m Normal Created pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Created container: istio-proxy 22m Normal Started pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Started container istio-proxy 22m Warning Unhealthy pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Readiness probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused 22m Warning ProbeError pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Readiness probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused... 22m Normal Pulled pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 21m Warning ProbeError pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused... 21m Warning Unhealthy pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused 21m Normal Killing pod/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x Stopping container payload-pre-processing 21m Normal Killing pod/e2e-derive-24d51017-openshift-default-57c46d584d-9x6j6 Stopping container istio-proxy 20m Normal AddedLabel gateway/e2e-mt-a5942927 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-a5942927 20m Normal SuccessfulCreate replicaset/e2e-mt-a5942927-openshift-default-6c596fb776 Created pod: e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft 20m Normal ScalingReplicaSet deployment/e2e-mt-a5942927-openshift-default Scaled up replica set e2e-mt-a5942927-openshift-default-6c596fb776 from 0 to 1 20m Normal AddedInterface pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Add eth0 [10.132.0.63/23] from ovn-kubernetes 20m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-a5942927 Scaled up replica set payload-processing-e2e-mt-a5942927-b5b9c4cc from 0 to 1 20m Normal AddedInterface pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Add eth0 [10.132.0.66/23] from ovn-kubernetes 20m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-a5942927 Scaled up replica set payload-pre-processing-e2e-mt-a5942927-64649cd7bb from 0 to 1 20m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-a5942927-64649cd7bb Created pod: payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 20m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-a5942927-64649cd7bb Error creating: pods "payload-pre-processing-e2e-mt-a5942927-64649cd7bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-a5942927: serviceaccount "payload-processing-e2e-mt-a5942927" not found 20m Normal Pulled pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 20m Normal AddedInterface pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Add eth0 [10.132.0.65/23] from ovn-kubernetes 20m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-a5942927-b5b9c4cc Created pod: payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n 20m Warning FailedCreate replicaset/payload-processing-e2e-mt-a5942927-b5b9c4cc Error creating: pods "payload-processing-e2e-mt-a5942927-b5b9c4cc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-a5942927: serviceaccount "payload-processing-e2e-mt-a5942927" not found 20m Normal Pulled pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 20m Normal Started pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Started container payload-processing 20m Normal Created pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Created container: payload-pre-processing 20m Normal Started pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Started container payload-pre-processing 20m Normal Created pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Created container: payload-processing 20m Normal Killing pod/payload-processing-e2e-inf-a-21d2a4-58d9f9c75c-7nxbd Stopping container payload-processing 20m Normal Killing pod/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm Stopping container payload-pre-processing 20m Normal Killing pod/e2e-inf-a-21d2a4-openshift-default-6447f5dd4b-p6c6q Stopping container istio-proxy 19m Normal Killing pod/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm Stopping container payload-pre-processing 19m Normal Killing pod/payload-pre-processing-e2e-mt-a5942927-64649cd7bb-jz2h7 Stopping container payload-pre-processing 19m Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-mt-a5942927 Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-mt-a5942927: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-mt-a5942927": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-mt-a5942927, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aff20bad-f62c-47d4-a18d-a870a356c023, UID in object meta: 19m Normal Killing pod/payload-processing-e2e-mt-a5942927-b5b9c4cc-rtq5n Stopping container payload-processing 19m Normal Killing pod/e2e-inf-b-e93f57-openshift-default-744dd4d998-6qh4q Stopping container istio-proxy 19m Normal Started pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Started container istio-proxy 19m Normal Created pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Created container: istio-proxy 19m Normal Pulled pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 19m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-50372a-openshift-default-57b557d8b Created pod: e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb 19m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-50372a-openshift-default Scaled up replica set e2e-shared-a-w2-50372a-openshift-default-57b557d8b from 0 to 1 19m Normal AddedLabel gateway/e2e-shared-a-w2-50372a Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-50372a 19m Normal AddedInterface pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Add eth0 [10.132.0.67/23] from ovn-kubernetes 19m Warning Unhealthy pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused 19m Warning ProbeError pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused... 19m Normal Killing pod/e2e-mt-a5942927-openshift-default-6c596fb776-xc2ft Stopping container istio-proxy 19m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf Error creating: pods "payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-50372a: serviceaccount "payload-processing-e2e-shared-a-w2-50372a" not found 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb Created pod: payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb Error creating: pods "payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-50372a: serviceaccount "payload-processing-e2e-shared-a-w2-50372a" not found 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-50372a Scaled up replica set payload-processing-e2e-shared-a-w2-50372a-8648d87bdf from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf Created pod: payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-50372a Scaled up replica set payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb from 0 to 1 19m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Started container payload-pre-processing 19m Normal Created pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Created container: payload-processing 19m Normal Started pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Started container payload-processing 19m Normal Pulled pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Add eth0 [10.132.0.70/23] from ovn-kubernetes 19m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Add eth0 [10.132.0.69/23] from ovn-kubernetes 19m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Created container: payload-pre-processing 19m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748 Created pod: e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 19m Normal AddedLabel gateway/e2e-shared-a-w1-c8e1cf Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-c8e1cf 19m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-c8e1cf-openshift-default Scaled up replica set e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748 from 0 to 1 19m Normal AddedInterface pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Add eth0 [10.133.0.60/23] from ovn-kubernetes 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687 Created pod: payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf: serviceaccount "payload-processing-e2e-shared-a-w1-c8e1cf" not found 19m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687 Error creating: pods "payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf: serviceaccount "payload-processing-e2e-shared-a-w1-c8e1cf" not found 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-c8e1cf Scaled up replica set payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8 from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8 Created pod: payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-c8e1cf Scaled up replica set payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687 from 0 to 1 19m Normal Started pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Started container payload-processing 19m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Add eth0 [10.134.0.58/23] from ovn-kubernetes 19m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Created container: payload-pre-processing 19m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Started container payload-pre-processing 19m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Add eth0 [10.134.0.57/23] from ovn-kubernetes 19m Normal Created pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Created container: payload-processing 19m Normal Pulled pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal AddedLabel gateway/e2e-shared-b-w2-f9c5c2 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-f9c5c2 19m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-f9c5c2-openshift-default Scaled up replica set e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb from 0 to 1 19m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb Created pod: e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h 19m Normal AddedInterface pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Add eth0 [10.132.0.72/23] from ovn-kubernetes 19m Normal Pulled pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 19m Normal Created pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Created container: istio-proxy 19m Normal Started pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Started container istio-proxy 19m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c Error creating: pods "payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2: serviceaccount "payload-processing-e2e-shared-b-w2-f9c5c2" not found 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b Error creating: pods "payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2: serviceaccount "payload-processing-e2e-shared-b-w2-f9c5c2" not found 19m Normal AddedLabel gateway/e2e-shared-b-w1-da587f Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-da587f 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c Created pod: payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-f9c5c2 Scaled up replica set payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c from 0 to 1 19m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-da587f-openshift-default Scaled up replica set e2e-shared-b-w1-da587f-openshift-default-7c48846ffb from 0 to 1 19m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb Created pod: e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-f9c5c2 Scaled up replica set payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b Created pod: payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp 19m Warning Unhealthy pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 19m Normal Created pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Created container: payload-processing 19m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Created container: payload-pre-processing 19m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Started container payload-pre-processing 19m Normal Started pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Started container payload-processing 19m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Add eth0 [10.134.0.60/23] from ovn-kubernetes 19m Warning ProbeError pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 19m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal AddedInterface pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Add eth0 [10.133.0.61/23] from ovn-kubernetes 19m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Add eth0 [10.134.0.59/23] from ovn-kubernetes 19m Normal Pulled pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-da587f Scaled up replica set payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8 Error creating: pods "payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-da587f: serviceaccount "payload-processing-e2e-shared-b-w1-da587f" not found 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8 Created pod: payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-da587f Scaled up replica set payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f from 0 to 1 18m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f Error creating: pods "payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-da587f: serviceaccount "payload-processing-e2e-shared-b-w1-da587f" not found 18m Normal Started pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Started container istio-proxy 18m Normal Created pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Created container: payload-processing 18m Normal Created pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Created container: istio-proxy 18m Normal Pulled pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Started container payload-pre-processing 18m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Created container: payload-pre-processing 18m Normal Pulled pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 18m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Add eth0 [10.134.0.61/23] from ovn-kubernetes 18m Normal Started pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Started container payload-processing 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f Created pod: payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq 18m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Add eth0 [10.134.0.62/23] from ovn-kubernetes 18m Warning Unhealthy pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused 18m Warning ProbeError pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused... 18m Normal AddedLabel gateway/e2e-mt-7cbcece2 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-7cbcece2 18m Normal SuccessfulCreate replicaset/e2e-mt-7cbcece2-openshift-default-9d5789cc6 Created pod: e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 18m Normal AddedInterface pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Add eth0 [10.133.0.66/23] from ovn-kubernetes 18m Normal ScalingReplicaSet deployment/e2e-mt-7cbcece2-openshift-default Scaled up replica set e2e-mt-7cbcece2-openshift-default-9d5789cc6 from 0 to 1 18m Normal Pulled pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 18m Normal Started pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Started container istio-proxy 18m Normal Created pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Created container: istio-proxy 18m Normal Pulled pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 18m Normal Created pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Created container: istio-proxy 18m Normal Started pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Started container istio-proxy 18m Warning ProbeError pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 18m Warning Unhealthy pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 18m Warning Unhealthy pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 18m Warning ProbeError pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674 Created pod: payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn 18m Normal Pulled pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Created pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Created container: payload-pre-processing 18m Normal AddedInterface pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Add eth0 [10.132.0.76/23] from ovn-kubernetes 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674 Error creating: pods "payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-7cbcece2: serviceaccount "payload-processing-e2e-mt-7cbcece2" not found 18m Normal Created pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Created container: payload-processing 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-7cbcece2-69b868bd6f Created pod: payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 18m Warning FailedCreate replicaset/payload-processing-e2e-mt-7cbcece2-69b868bd6f Error creating: pods "payload-processing-e2e-mt-7cbcece2-69b868bd6f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-7cbcece2: serviceaccount "payload-processing-e2e-mt-7cbcece2" not found 18m Normal AddedInterface pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Add eth0 [10.134.0.63/23] from ovn-kubernetes 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-7cbcece2 Scaled up replica set payload-pre-processing-e2e-mt-7cbcece2-5d67c65674 from 0 to 1 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-7cbcece2 Scaled up replica set payload-processing-e2e-mt-7cbcece2-69b868bd6f from 0 to 1 18m Normal Pulled pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Started pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Started container payload-pre-processing 18m Normal Started pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Started container payload-processing 17m Normal Started pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Started container istio-proxy 17m Normal Created pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Created container: istio-proxy 17m Normal Pulled pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 17m Warning Unhealthy pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 17m Warning ProbeError pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 17m Normal Killing pod/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx Stopping container payload-processing 17m Normal Killing pod/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d Stopping container payload-pre-processing 17m Normal Killing pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Stopping container istio-proxy 17m Warning ProbeError pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Readiness probe error: HTTP probe failed with statuscode: 503... 17m Warning Unhealthy pod/e2e-worker-w3-33737b-openshift-default-58b799f4b4-lnxhf Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Killing pod/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz Stopping container payload-processing 16m Normal Killing pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Stopping container istio-proxy 16m Warning ProbeError pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Readiness probe error: HTTP probe failed with statuscode: 503... 16m Warning Unhealthy pod/e2e-worker-w0-15949d-openshift-default-7b5f7fcf76-p9zw7 Readiness probe failed: HTTP probe failed with statuscode: 503 16m Normal Killing pod/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2 Stopping container payload-processing 16m Normal Killing pod/payload-pre-processing-e2e-mt-7cbcece2-5d67c65674-487kn Stopping container payload-pre-processing 15m Normal Killing pod/e2e-mt-7cbcece2-openshift-default-9d5789cc6-h4sd7 Stopping container istio-proxy 14m Normal SuccessfulCreate replicaset/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd Created pod: e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh 14m Normal AddedLabel gateway/e2e-api-a-88b9ff Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-88b9ff 14m Normal ScalingReplicaSet deployment/e2e-api-a-88b9ff-openshift-default Scaled up replica set e2e-api-a-88b9ff-openshift-default-7c6f4c48cd from 0 to 1 14m Normal AddedInterface pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Add eth0 [10.133.0.74/23] from ovn-kubernetes 14m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-88b9ff Scaled up replica set payload-processing-e2e-api-a-88b9ff-5d964f9597 from 0 to 1 14m Normal Started pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Started container payload-pre-processing 14m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-88b9ff-5d964f9597 Created pod: payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 14m Normal AddedInterface pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Add eth0 [10.134.0.68/23] from ovn-kubernetes 14m Warning FailedCreate replicaset/payload-processing-e2e-api-a-88b9ff-5d964f9597 Error creating: pods "payload-processing-e2e-api-a-88b9ff-5d964f9597-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-88b9ff: serviceaccount "payload-processing-e2e-api-a-88b9ff" not found 14m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-88b9ff Scaled up replica set payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b from 0 to 1 14m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b Created pod: payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 14m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b Error creating: pods "payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-88b9ff: serviceaccount "payload-processing-e2e-api-a-88b9ff" not found 14m Normal Pulled pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 14m Normal Created pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Created container: payload-processing 14m Normal Created pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Created container: payload-pre-processing 14m Normal Pulled pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 14m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Add eth0 [10.134.0.67/23] from ovn-kubernetes 14m Normal Started pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Started container payload-processing 13m Normal Pulled pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 13m Normal Created pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Created container: istio-proxy 13m Normal Started pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Started container istio-proxy 13m Warning ProbeError pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Startup probe error: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Startup probe failed: Get "http://10.133.0.74:15021/healthz/ready": dial tcp 10.133.0.74:15021: connect: connection refused 13m Normal SuccessfulCreate replicaset/e2e-api-b-26cbe6-openshift-default-5d4d8895db Created pod: e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r 13m Normal ScalingReplicaSet deployment/e2e-api-b-26cbe6-openshift-default Scaled up replica set e2e-api-b-26cbe6-openshift-default-5d4d8895db from 0 to 1 13m Normal AddedLabel gateway/e2e-api-b-26cbe6 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-26cbe6 13m Normal AddedInterface pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Add eth0 [10.133.0.76/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-26cbe6-6f86965f89 Created pod: payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-26cbe6 Scaled up replica set payload-pre-processing-e2e-api-b-26cbe6-689d748786 from 0 to 1 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-26cbe6-689d748786 Error creating: pods "payload-pre-processing-e2e-api-b-26cbe6-689d748786-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-26cbe6: serviceaccount "payload-processing-e2e-api-b-26cbe6" not found 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-26cbe6-689d748786 Created pod: payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw 13m Warning FailedCreate replicaset/payload-processing-e2e-api-b-26cbe6-6f86965f89 Error creating: pods "payload-processing-e2e-api-b-26cbe6-6f86965f89-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-26cbe6: serviceaccount "payload-processing-e2e-api-b-26cbe6" not found 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-26cbe6 Scaled up replica set payload-processing-e2e-api-b-26cbe6-6f86965f89 from 0 to 1 13m Normal Started pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Started container payload-pre-processing 13m Normal Created pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Created container: payload-pre-processing 13m Normal Started pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Started container payload-processing 13m Normal Created pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Created container: payload-processing 13m Normal Pulled pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 13m Normal Pulled pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 13m Normal AddedInterface pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Add eth0 [10.134.0.70/23] from ovn-kubernetes 13m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Add eth0 [10.132.0.77/23] from ovn-kubernetes 13m Normal Created pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Created container: istio-proxy 13m Normal Pulled pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 13m Normal Started pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Started container istio-proxy 13m Warning ProbeError pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Startup probe error: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Startup probe failed: Get "http://10.133.0.76:15021/healthz/ready": dial tcp 10.133.0.76:15021: connect: connection refused 12m Normal Killing pod/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59 Stopping container payload-processing 12m Normal Killing pod/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84 Stopping container payload-pre-processing 12m Normal Killing pod/e2e-api-a-88b9ff-openshift-default-7c6f4c48cd-hjclh Stopping container istio-proxy 12m Normal Killing pod/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb Stopping container payload-processing 12m Normal Killing pod/payload-pre-processing-e2e-api-b-26cbe6-689d748786-92mtw Stopping container payload-pre-processing 12m Normal Killing pod/e2e-api-b-26cbe6-openshift-default-5d4d8895db-5x85r Stopping container istio-proxy 12m Normal Killing pod/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4 Stopping container payload-processing 12m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj Stopping container payload-pre-processing 12m Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-shared-a-w1-c8e1cf Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-shared-a-w1-c8e1cf": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 20331135-8a43-448a-93d4-f0e7ce74f902, UID in object meta: 12m Normal Killing pod/e2e-shared-a-w1-c8e1cf-openshift-default-6dcbc66748-wf8l9 Stopping container istio-proxy 11m Normal Killing pod/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq Stopping container payload-processing 11m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf Stopping container payload-pre-processing 11m Normal Killing pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Stopping container istio-proxy 11m Warning Unhealthy pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Readiness probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 11m Warning ProbeError pod/e2e-shared-b-w1-da587f-openshift-default-7c48846ffb-2ntqf Readiness probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 11m Normal AddedLabel gateway/e2e-ipp-a-521326 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-521326 11m Normal ScalingReplicaSet deployment/e2e-ipp-a-521326-openshift-default Scaled up replica set e2e-ipp-a-521326-openshift-default-5ddc57db88 from 0 to 1 11m Normal SuccessfulCreate replicaset/e2e-ipp-a-521326-openshift-default-5ddc57db88 Created pod: e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c 11m Normal AddedInterface pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Add eth0 [10.133.0.84/23] from ovn-kubernetes 11m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-521326 Scaled up replica set payload-processing-e2e-ipp-a-521326-79cf58d6c from 0 to 1 11m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-521326 Scaled up replica set payload-pre-processing-e2e-ipp-a-521326-7655f5868b from 0 to 1 11m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-521326-7655f5868b Error creating: pods "payload-pre-processing-e2e-ipp-a-521326-7655f5868b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-521326: serviceaccount "payload-processing-e2e-ipp-a-521326" not found 11m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-521326-79cf58d6c Error creating: pods "payload-processing-e2e-ipp-a-521326-79cf58d6c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-521326: serviceaccount "payload-processing-e2e-ipp-a-521326" not found 11m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 11m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Add eth0 [10.134.0.73/23] from ovn-kubernetes 11m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-521326-7655f5868b Created pod: payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn 11m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-521326-79cf58d6c Created pod: payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv 11m Normal Started pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Started container payload-pre-processing 11m Normal Started pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Started container payload-processing 11m Normal Created pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Created container: payload-processing 11m Normal Pulled pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 11m Normal AddedInterface pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Add eth0 [10.134.0.74/23] from ovn-kubernetes 11m Normal Created pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Created container: payload-pre-processing 11m Warning Unhealthy pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Readiness probe failed: Get "http://10.133.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Readiness probe error: Get "http://10.133.0.84:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Normal Created pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Created container: istio-proxy 11m Normal Pulled pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 11m Normal Started pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Started container istio-proxy 11m Warning Unhealthy pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Startup probe failed: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused 11m Warning ProbeError pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Startup probe error: Get "http://10.133.0.84:15021/healthz/ready": dial tcp 10.133.0.84:15021: connect: connection refused... 11m Normal ScalingReplicaSet deployment/e2e-ipp-b-30ef69-openshift-default Scaled up replica set e2e-ipp-b-30ef69-openshift-default-74c757bf7d from 0 to 1 11m Normal SuccessfulCreate replicaset/e2e-ipp-b-30ef69-openshift-default-74c757bf7d Created pod: e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 11m Normal AddedLabel gateway/e2e-ipp-b-30ef69 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-30ef69 11m Normal AddedInterface pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Add eth0 [10.133.0.85/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-30ef69-78b476ddb5 Created pod: payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 10m Normal Pulled pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-30ef69 Scaled up replica set payload-processing-e2e-ipp-b-30ef69-78b476ddb5 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-30ef69 Scaled up replica set payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6 from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6 Created pod: payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6 Error creating: pods "payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-30ef69: serviceaccount "payload-processing-e2e-ipp-b-30ef69" not found 10m Normal Started pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Started container payload-pre-processing 10m Normal Created pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Created container: payload-pre-processing 10m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Add eth0 [10.134.0.76/23] from ovn-kubernetes 10m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-30ef69-78b476ddb5 Error creating: pods "payload-processing-e2e-ipp-b-30ef69-78b476ddb5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-30ef69: serviceaccount "payload-processing-e2e-ipp-b-30ef69" not found 10m Normal Started pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Started container payload-processing 10m Normal AddedInterface pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Add eth0 [10.134.0.77/23] from ovn-kubernetes 10m Normal Created pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Created container: payload-processing 10m Warning Unhealthy pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Readiness probe failed: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 10m Warning ProbeError pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Readiness probe error: Get "http://10.133.0.85:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 10m Normal Pulled pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 10m Normal Created pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Created container: istio-proxy 10m Normal Started pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Started container istio-proxy 10m Warning Unhealthy pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Startup probe failed: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused 10m Warning ProbeError pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Startup probe error: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused... 9m25s Normal AddedLabel gateway/e2e-ipp-cleanup-fa8af0 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-cleanup-fa8af0 9m24s Normal AddedInterface pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Add eth0 [10.133.0.91/23] from ovn-kubernetes 9m24s Normal ScalingReplicaSet deployment/e2e-ipp-cleanup-fa8af0-openshift-default Scaled up replica set e2e-ipp-cleanup-fa8af0-openshift-default-748c577596 from 0 to 1 9m24s Normal SuccessfulCreate replicaset/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596 Created pod: e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj 9m18s Warning FailedCreate replicaset/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c Error creating: pods "payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0: serviceaccount "payload-processing-e2e-ipp-cleanup-fa8af0" not found 9m18s Normal Pulled pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 9m18s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-cleanup-fa8af0 Scaled up replica set payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6 from 0 to 1 9m18s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6 Created pod: payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n 9m18s Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6 Error creating: pods "payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0: serviceaccount "payload-processing-e2e-ipp-cleanup-fa8af0" not found 9m18s Normal AddedInterface pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Add eth0 [10.134.0.79/23] from ovn-kubernetes 9m18s Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-cleanup-fa8af0 Scaled up replica set payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c from 0 to 1 9m18s Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c Created pod: payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj 9m17s Normal AddedInterface pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Add eth0 [10.134.0.80/23] from ovn-kubernetes 9m17s Normal Started pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Started container payload-processing 9m17s Normal Started pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Started container payload-pre-processing 9m17s Normal Pulled pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 9m17s Normal Created pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Created container: payload-pre-processing 9m17s Normal Created pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Created container: payload-processing 9m2s Normal Created pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Created container: istio-proxy 9m2s Normal Pulled pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 9m2s Normal Started pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Started container istio-proxy 8m56s Warning Unhealthy pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Startup probe failed: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused 8m56s Warning ProbeError pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Startup probe error: Get "http://10.133.0.91:15021/healthz/ready": dial tcp 10.133.0.91:15021: connect: connection refused... 8m40s Normal Killing pod/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj Stopping container payload-processing 8m40s Normal Killing pod/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n Stopping container payload-pre-processing 8m33s Normal Killing pod/e2e-ipp-cleanup-fa8af0-openshift-default-748c577596-w4nmj Stopping container istio-proxy 8m14s Normal Killing pod/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn Stopping container payload-pre-processing 8m14s Normal Killing pod/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv Stopping container payload-processing 7m26s Normal Killing pod/e2e-ipp-a-521326-openshift-default-5ddc57db88-b5w4c Stopping container istio-proxy 7m15s Normal Killing pod/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw Stopping container payload-pre-processing 7m15s Normal Killing pod/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6 Stopping container payload-processing 7m8s Normal Killing pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Stopping container istio-proxy 7m3s Warning ProbeError pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Readiness probe error: HTTP probe failed with statuscode: 503... 7m3s Warning Unhealthy pod/e2e-ipp-b-30ef69-openshift-default-74c757bf7d-mjpb4 Readiness probe failed: HTTP probe failed with statuscode: 503 6m57s Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-shared-a-w2-50372a Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-shared-a-w2-50372a: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-shared-a-w2-50372a": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-shared-a-w2-50372a, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 76ea6d61-ef66-4f91-ba7d-ff56e5c423db, UID in object meta: 6m57s Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-50372a-84d8f46cbb-xr47t Stopping container payload-pre-processing 6m57s Normal Killing pod/payload-processing-e2e-shared-a-w2-50372a-8648d87bdf-g7hjz Stopping container payload-processing 6m51s Normal Killing pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Stopping container istio-proxy 6m47s Warning ProbeError pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Readiness probe error: HTTP probe failed with statuscode: 503... 6m47s Warning Unhealthy pod/e2e-shared-a-w2-50372a-openshift-default-57b557d8b-65zcb Readiness probe failed: HTTP probe failed with statuscode: 503 6m40s Normal Killing pod/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp Stopping container payload-processing 6m40s Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp Stopping container payload-pre-processing 6m33s Normal Killing pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Stopping container istio-proxy 6m29s Warning ProbeError pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Readiness probe error: HTTP probe failed with statuscode: 503... 6m29s Warning Unhealthy pod/e2e-shared-b-w2-f9c5c2-openshift-default-7765c9c9fb-m997h Readiness probe failed: HTTP probe failed with statuscode: 503