=== events in openshift-ingress (2026-09-19T11:11:48Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 18m Normal Scheduled pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd to ip-10-0-143-186.ec2.internal 32m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 to ip-10-0-143-38.ec2.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 to ip-10-0-143-186.ec2.internal 19m Normal Scheduled pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Successfully assigned openshift-ingress/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Successfully assigned openshift-ingress/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 to ip-10-0-143-38.ec2.internal 9m1s Normal Scheduled pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Successfully assigned openshift-ingress/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 to ip-10-0-143-186.ec2.internal 10m Normal Scheduled pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j to ip-10-0-143-38.ec2.internal 10m Normal Scheduled pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb to ip-10-0-143-38.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k to ip-10-0-143-38.ec2.internal 32m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd to ip-10-0-143-38.ec2.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp to ip-10-0-143-186.ec2.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn to ip-10-0-143-38.ec2.internal 19m Normal Scheduled pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Successfully assigned openshift-ingress/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr to ip-10-0-143-186.ec2.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Successfully assigned openshift-ingress/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw to ip-10-0-143-38.ec2.internal 18m Normal Scheduled pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Successfully assigned openshift-ingress/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d to ip-10-0-143-38.ec2.internal 9m1s Normal Scheduled pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz to ip-10-0-143-186.ec2.internal 22m Normal Scheduled pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Successfully assigned openshift-ingress/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 to ip-10-0-143-38.ec2.internal 13m Normal Scheduled pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Successfully assigned openshift-ingress/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 to ip-10-0-143-186.ec2.internal 22m Normal Scheduled pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-76f657cdc4-dkm2p Successfully assigned openshift-ingress/payload-pre-processing-76f657cdc4-dkm2p to ip-10-0-143-38.ec2.internal 18m Normal Scheduled pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Successfully assigned openshift-ingress/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Successfully assigned openshift-ingress/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Successfully assigned openshift-ingress/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr to ip-10-0-143-38.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Successfully assigned openshift-ingress/payload-processing-e2e-ait-963aea34-5df966b875-fxngj to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Successfully assigned openshift-ingress/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr to ip-10-0-143-38.ec2.internal 19m Normal Scheduled pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb to ip-10-0-143-38.ec2.internal 17m Normal Scheduled pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk to ip-10-0-143-186.ec2.internal 9m8s Normal Scheduled pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Successfully assigned openshift-ingress/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g to ip-10-0-143-186.ec2.internal 10m Normal Scheduled pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Successfully assigned openshift-ingress/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/payload-processing-85fd65c94c-7hp29 Successfully assigned openshift-ingress/payload-processing-85fd65c94c-7hp29 to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 to ip-10-0-143-186.ec2.internal 32m Normal Scheduled pod/istiod-openshift-gateway-7b7995d5cd-slfdj Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-slfdj to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 to ip-10-0-143-38.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl to ip-10-0-143-38.ec2.internal 13m Normal Scheduled pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Successfully assigned openshift-ingress/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Successfully assigned openshift-ingress/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Successfully assigned openshift-ingress/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb to ip-10-0-143-186.ec2.internal 32m Normal Scheduled pod/kube-auth-proxy-8b786b9b5-wppgk Successfully assigned openshift-ingress/kube-auth-proxy-8b786b9b5-wppgk to ip-10-0-140-105.ec2.internal 29m Normal Scheduled pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-g4xvk to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Successfully assigned openshift-ingress/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv to ip-10-0-143-38.ec2.internal 23m Normal Scheduled pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr to ip-10-0-143-186.ec2.internal 24m Normal Scheduled pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Successfully assigned openshift-ingress/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 to ip-10-0-143-38.ec2.internal 18m Normal Scheduled pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Successfully assigned openshift-ingress/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j to ip-10-0-143-38.ec2.internal 13m Normal Scheduled pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Successfully assigned openshift-ingress/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq to ip-10-0-143-38.ec2.internal 17m Normal Scheduled pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Successfully assigned openshift-ingress/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl to ip-10-0-143-186.ec2.internal 28m Normal Scheduled pod/payload-processing-5b54b7fbb7-4nssc Successfully assigned openshift-ingress/payload-processing-5b54b7fbb7-4nssc to ip-10-0-143-38.ec2.internal 28m Normal Scheduled pod/payload-pre-processing-59fbc47bf8-78572 Successfully assigned openshift-ingress/payload-pre-processing-59fbc47bf8-78572 to ip-10-0-143-186.ec2.internal 17m Normal Scheduled pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Successfully assigned openshift-ingress/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk to ip-10-0-143-38.ec2.internal 36m Normal Scheduled pod/router-default-9998785cb-l2lkw Successfully assigned openshift-ingress/router-default-9998785cb-l2lkw to ip-10-0-143-38.ec2.internal 18m Normal Scheduled pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb to ip-10-0-143-38.ec2.internal 22m Normal Scheduled pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Successfully assigned openshift-ingress/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr to ip-10-0-143-38.ec2.internal 24m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq to ip-10-0-143-186.ec2.internal 18m Normal Scheduled pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh to ip-10-0-143-38.ec2.internal 12m Normal Scheduled pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Successfully assigned openshift-ingress/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 to ip-10-0-143-186.ec2.internal 22m Normal Scheduled pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Successfully assigned openshift-ingress/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg to ip-10-0-143-38.ec2.internal 10m Normal Scheduled pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Successfully assigned openshift-ingress/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq to ip-10-0-143-186.ec2.internal 31m Normal Scheduled pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-mjlb7 to ip-10-0-143-38.ec2.internal 32m Normal Scheduled pod/kube-auth-proxy-8b786b9b5-h9glx Successfully assigned openshift-ingress/kube-auth-proxy-8b786b9b5-h9glx to ip-10-0-143-186.ec2.internal 43m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-9998785cb from 0 to 1 43m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 43m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 43m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 42m Warning FailedCreate replicaset/router-default-9998785cb Error creating: pods "router-default-9998785cb-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 39m Warning FailedCreate replicaset/router-default-9998785cb Error creating: pods "router-default-9998785cb-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 37m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 36m Normal SuccessfulCreate replicaset/router-default-9998785cb Created pod: router-default-9998785cb-l2lkw 36m Warning FailedMount pod/router-default-9998785cb-l2lkw MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 36m Warning FailedMount pod/router-default-9998785cb-l2lkw MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 36m Normal Started pod/router-default-9998785cb-l2lkw Started container router 36m Normal Created pod/router-default-9998785cb-l2lkw Created container: router 36m Normal Pulled pod/router-default-9998785cb-l2lkw Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f" already present on machine 36m Normal AddedInterface pod/router-default-9998785cb-l2lkw Add eth0 [10.132.0.15/23] from ovn-kubernetes 32m Warning FailedCreate replicaset/kube-auth-proxy-8b786b9b5 Error creating: pods "kube-auth-proxy-8b786b9b5-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found 32m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-8b786b9b5 from 0 to 2 32m Normal SuccessfulCreate replicaset/kube-auth-proxy-8b786b9b5 Created pod: kube-auth-proxy-8b786b9b5-wppgk 32m Normal SuccessfulCreate replicaset/kube-auth-proxy-8b786b9b5 Created pod: kube-auth-proxy-8b786b9b5-h9glx 32m Normal AddedInterface pod/kube-auth-proxy-8b786b9b5-wppgk Add eth0 [10.134.0.26/23] from ovn-kubernetes 32m Normal Pulling pod/kube-auth-proxy-8b786b9b5-wppgk Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 32m Normal AddedInterface pod/kube-auth-proxy-8b786b9b5-h9glx Add eth0 [10.133.0.21/23] from ovn-kubernetes 32m Normal Pulling pod/kube-auth-proxy-8b786b9b5-h9glx Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 32m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7b7995d5cd Created pod: istiod-openshift-gateway-7b7995d5cd-slfdj 32m Normal AddedInterface pod/istiod-openshift-gateway-7b7995d5cd-slfdj Add eth0 [10.133.0.22/23] from ovn-kubernetes 32m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1 32m Normal Pulled pod/kube-auth-proxy-8b786b9b5-h9glx Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.715s (2.715s including waiting). Image size: 265120132 bytes. 32m Normal Created pod/kube-auth-proxy-8b786b9b5-h9glx Created container: kube-auth-proxy 32m Normal Started pod/kube-auth-proxy-8b786b9b5-h9glx Started container kube-auth-proxy 32m Normal Pulling pod/istiod-openshift-gateway-7b7995d5cd-slfdj Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" 32m Normal Started pod/kube-auth-proxy-8b786b9b5-wppgk Started container kube-auth-proxy 32m Normal Created pod/kube-auth-proxy-8b786b9b5-wppgk Created container: kube-auth-proxy 32m Normal Pulled pod/kube-auth-proxy-8b786b9b5-wppgk Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 3.138s (3.138s including waiting). Image size: 265120132 bytes. 32m Normal Pulled pod/istiod-openshift-gateway-7b7995d5cd-slfdj Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.417s (2.417s including waiting). Image size: 181126104 bytes. 32m Normal Created pod/istiod-openshift-gateway-7b7995d5cd-slfdj Created container: discovery 32m Normal Started pod/istiod-openshift-gateway-7b7995d5cd-slfdj Started container discovery 32m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Add eth0 [10.132.0.28/23] from ovn-kubernetes 32m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-d66cd8c7c Created pod: data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd 32m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Add eth0 [10.132.0.27/23] from ovn-kubernetes 32m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-d66cd8c7c from 0 to 1 32m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-6745bb66bc from 0 to 1 32m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-6745bb66bc Created pod: data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 32m Normal Pulling pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 32m Normal Pulling pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 32m Normal Created pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Created container: istio-proxy 32m Normal Pulled pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.662s (2.662s including waiting). Image size: 181318953 bytes. 32m Normal Created pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Created container: istio-proxy 32m Normal Started pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-tnzgd Started container istio-proxy 32m Normal Started pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Started container istio-proxy 32m Normal Pulled pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.71s (2.71s including waiting). Image size: 181318953 bytes. 32m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled down replica set data-science-gateway-data-science-gateway-class-6745bb66bc from 1 to 0 32m Normal SuccessfulDelete replicaset/data-science-gateway-data-science-gateway-class-6745bb66bc Deleted pod: data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 32m Warning ProbeError pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Startup probe error: Get "http://10.132.0.27:15021/healthz/ready": dial tcp 10.132.0.27:15021: connect: connection refused... 32m Warning Unhealthy pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Startup probe failed: Get "http://10.132.0.27:15021/healthz/ready": dial tcp 10.132.0.27:15021: connect: connection refused 32m Normal Killing pod/data-science-gateway-data-science-gateway-class-6745bb66bcrq4b6 Stopping container istio-proxy 31m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1 31m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-5677ccb49b Created pod: istiod-openshift-gateway-5677ccb49b-mjlb7 31m Normal AddedInterface pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Add eth0 [10.132.0.30/23] from ovn-kubernetes 31m Normal Pulling pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" 31m Normal Created pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Created container: discovery 31m Normal Started pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Started container discovery 31m Normal Pulled pod/istiod-openshift-gateway-5677ccb49b-mjlb7 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.468s (2.468s including waiting). Image size: 181126104 bytes. 31m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0 31m Normal Killing pod/istiod-openshift-gateway-7b7995d5cd-slfdj Stopping container discovery 31m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7b7995d5cd Deleted pod: istiod-openshift-gateway-7b7995d5cd-slfdj 31m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 31m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 31m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 29m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-78bb78ff78 from 0 to 1 29m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 29m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-78bb78ff78 Created pod: maas-default-gateway-openshift-default-78bb78ff78-g4xvk 29m Normal AddedInterface pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Add eth0 [10.133.0.33/23] from ovn-kubernetes 29m Normal Pulling pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 29m Normal Pulled pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.438s (2.438s including waiting). Image size: 181318953 bytes. 28m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5b54b7fbb7 from 0 to 1 28m Warning FailedCreate replicaset/payload-pre-processing-59fbc47bf8 Error creating: pods "payload-pre-processing-59fbc47bf8-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 28m Warning FailedCreate replicaset/payload-processing-5b54b7fbb7 Error creating: pods "payload-processing-5b54b7fbb7-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 28m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-59fbc47bf8 from 0 to 1 28m Normal Pulling pod/payload-pre-processing-59fbc47bf8-78572 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 28m Normal AddedInterface pod/payload-processing-5b54b7fbb7-4nssc Add eth0 [10.132.0.32/23] from ovn-kubernetes 28m Normal SuccessfulCreate replicaset/payload-processing-5b54b7fbb7 Created pod: payload-processing-5b54b7fbb7-4nssc 28m Normal AddedInterface pod/payload-pre-processing-59fbc47bf8-78572 Add eth0 [10.133.0.43/23] from ovn-kubernetes 28m Normal Pulling pod/payload-processing-5b54b7fbb7-4nssc Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 28m Normal SuccessfulCreate replicaset/payload-pre-processing-59fbc47bf8 Created pod: payload-pre-processing-59fbc47bf8-78572 28m Normal Pulled pod/payload-processing-5b54b7fbb7-4nssc Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.299s (2.299s including waiting). Image size: 166453408 bytes. 28m Normal Pulled pod/payload-pre-processing-59fbc47bf8-78572 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.207s (2.207s including waiting). Image size: 166453408 bytes. 28m Normal Created pod/payload-pre-processing-59fbc47bf8-78572 Created container: payload-pre-processing 28m Normal Started pod/payload-pre-processing-59fbc47bf8-78572 Started container payload-pre-processing 28m Normal Created pod/payload-processing-5b54b7fbb7-4nssc Created container: payload-processing 28m Normal Started pod/payload-processing-5b54b7fbb7-4nssc Started container payload-processing 28m Normal Created pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Created container: istio-proxy 28m Normal Pulled pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 28m Normal Started pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Started container istio-proxy 28m Warning Unhealthy pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused 28m Warning ProbeError pod/maas-default-gateway-openshift-default-78bb78ff78-g4xvk Startup probe error: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused... 25m Normal Killing pod/payload-pre-processing-59fbc47bf8-78572 Stopping container payload-pre-processing 25m Normal Killing pod/payload-processing-5b54b7fbb7-4nssc Stopping container payload-processing 24m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-76f657cdc4 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-85fd65c94c Created pod: payload-processing-85fd65c94c-7hp29 24m Normal Pulling pod/payload-processing-85fd65c94c-7hp29 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 24m Normal AddedInterface pod/payload-processing-85fd65c94c-7hp29 Add eth0 [10.133.0.49/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/payload-pre-processing-76f657cdc4 Created pod: payload-pre-processing-76f657cdc4-dkm2p 24m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-85fd65c94c from 0 to 1 24m Normal AddedInterface pod/payload-pre-processing-76f657cdc4-dkm2p Add eth0 [10.132.0.35/23] from ovn-kubernetes 24m Normal Pulling pod/payload-pre-processing-76f657cdc4-dkm2p Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 24m Normal Pulled pod/payload-processing-85fd65c94c-7hp29 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 1.069s (1.069s including waiting). Image size: 141049838 bytes. 24m Normal Started pod/payload-processing-85fd65c94c-7hp29 Started container payload-processing 24m Normal Created pod/payload-processing-85fd65c94c-7hp29 Created container: payload-processing 24m Normal Started pod/payload-pre-processing-76f657cdc4-dkm2p Started container payload-pre-processing 24m Normal Created pod/payload-pre-processing-76f657cdc4-dkm2p Created container: payload-pre-processing 24m Normal Pulled pod/payload-pre-processing-76f657cdc4-dkm2p Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.079s (2.079s including waiting). Image size: 141049838 bytes. 24m Normal AddedLabel gateway/e2e-worker-w0-cc1b19 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-cc1b19 24m Normal AddedLabel gateway/e2e-inf-a-f91f97 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-f91f97 24m Normal AddedLabel gateway/e2e-worker-w3-580e01 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-580e01 24m Normal SuccessfulCreate replicaset/e2e-inf-a-f91f97-openshift-default-7dc8555bfd Created pod: e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 24m Normal AddedInterface pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Add eth0 [10.133.0.50/23] from ovn-kubernetes 24m Normal ScalingReplicaSet deployment/e2e-inf-a-f91f97-openshift-default Scaled up replica set e2e-inf-a-f91f97-openshift-default-7dc8555bfd from 0 to 1 24m Normal SuccessfulCreate replicaset/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d Created pod: e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb 24m Normal ScalingReplicaSet deployment/e2e-worker-w0-cc1b19-openshift-default Scaled up replica set e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d from 0 to 1 24m Normal ScalingReplicaSet deployment/e2e-worker-w3-580e01-openshift-default Scaled up replica set e2e-worker-w3-580e01-openshift-default-db458d66d from 0 to 1 24m Normal AddedInterface pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Add eth0 [10.132.0.36/23] from ovn-kubernetes 24m Normal AddedInterface pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Add eth0 [10.132.0.37/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/e2e-worker-w3-580e01-openshift-default-db458d66d Created pod: e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-a-f91f97 Scaled up replica set payload-processing-e2e-inf-a-f91f97-597bb7c965 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-a-f91f97-597bb7c965 Created pod: payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-a-f91f97 Scaled up replica set payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4 from 0 to 1 24m Warning FailedCreate replicaset/payload-processing-e2e-inf-a-f91f97-597bb7c965 Error creating: pods "payload-processing-e2e-inf-a-f91f97-597bb7c965-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f91f97: serviceaccount "payload-processing-e2e-inf-a-f91f97" not found 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4 Created pod: payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4 Error creating: pods "payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-f91f97: serviceaccount "payload-processing-e2e-inf-a-f91f97" not found 24m Normal Created pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Created container: payload-pre-processing 24m Normal Pulled pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal AddedInterface pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Add eth0 [10.133.0.52/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-580e01-6885674c9 Created pod: payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-580e01 Scaled up replica set payload-processing-e2e-worker-w3-580e01-6885674c9 from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-580e01 Scaled up replica set payload-pre-processing-e2e-worker-w3-580e01-84f778859d from 0 to 1 24m Normal AddedInterface pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Add eth0 [10.132.0.38/23] from ovn-kubernetes 24m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-580e01-6885674c9 Error creating: pods "payload-processing-e2e-worker-w3-580e01-6885674c9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-580e01: serviceaccount "payload-processing-e2e-worker-w3-580e01" not found 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-580e01-84f778859d Created pod: payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-580e01-84f778859d Error creating: pods "payload-pre-processing-e2e-worker-w3-580e01-84f778859d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-580e01: serviceaccount "payload-processing-e2e-worker-w3-580e01" not found 24m Normal Started pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Started container payload-pre-processing 24m Normal Created pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Created container: payload-processing 24m Normal Started pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Started container payload-processing 24m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-cc1b19 Scaled up replica set payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b from 0 to 1 24m Normal Started pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Started container payload-processing 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b Error creating: pods "payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-cc1b19: serviceaccount "payload-processing-e2e-worker-w0-cc1b19" not found 24m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Add eth0 [10.132.0.39/23] from ovn-kubernetes 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b Created pod: payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq 24m Normal Created pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Created container: payload-pre-processing 24m Normal Started pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Started container payload-pre-processing 24m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-cc1b19-d96b479d9 Error creating: pods "payload-processing-e2e-worker-w0-cc1b19-d96b479d9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-cc1b19: serviceaccount "payload-processing-e2e-worker-w0-cc1b19" not found 24m Normal Created pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Created container: payload-processing 24m Normal Pulled pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal AddedInterface pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Add eth0 [10.132.0.40/23] from ovn-kubernetes 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-cc1b19 Scaled up replica set payload-processing-e2e-worker-w0-cc1b19-d96b479d9 from 0 to 1 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-cc1b19-d96b479d9 Created pod: payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd 24m Normal Created pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Created container: payload-pre-processing 24m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal AddedInterface pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Add eth0 [10.133.0.55/23] from ovn-kubernetes 24m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Add eth0 [10.133.0.54/23] from ovn-kubernetes 24m Normal Pulled pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal Created pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Created container: payload-processing 24m Normal Started pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Started container payload-pre-processing 24m Normal Started pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Started container payload-processing 24m Normal AddedLabel gateway/e2e-ait-b21bfa16 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-b21bfa16 24m Normal ScalingReplicaSet deployment/e2e-ait-b21bfa16-openshift-default Scaled up replica set e2e-ait-b21bfa16-openshift-default-65d6c97bc4 from 0 to 1 24m Normal SuccessfulCreate replicaset/e2e-ait-b21bfa16-openshift-default-65d6c97bc4 Created pod: e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr 24m Normal AddedInterface pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Add eth0 [10.132.0.41/23] from ovn-kubernetes 24m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-b21bfa16-5849646f46 Error creating: pods "payload-pre-processing-e2e-ait-b21bfa16-5849646f46-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b21bfa16: serviceaccount "payload-processing-e2e-ait-b21bfa16" not found 24m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-b21bfa16-7ddc596d8f Created pod: payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr 24m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-b21bfa16-5849646f46 Created pod: payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz 24m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-b21bfa16 Scaled up replica set payload-processing-e2e-ait-b21bfa16-7ddc596d8f from 0 to 1 24m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-b21bfa16 Scaled up replica set payload-pre-processing-e2e-ait-b21bfa16-5849646f46 from 0 to 1 24m Warning FailedCreate replicaset/payload-processing-e2e-ait-b21bfa16-7ddc596d8f Error creating: pods "payload-processing-e2e-ait-b21bfa16-7ddc596d8f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-b21bfa16: serviceaccount "payload-processing-e2e-ait-b21bfa16" not found 24m Normal Pulled pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal AddedInterface pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Add eth0 [10.133.0.56/23] from ovn-kubernetes 24m Normal Created pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Created container: payload-processing 24m Normal Started pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Started container payload-processing 24m Normal Started pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Started container payload-pre-processing 24m Normal Created pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Created container: payload-pre-processing 24m Normal Pulled pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 24m Normal AddedInterface pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Add eth0 [10.132.0.42/23] from ovn-kubernetes 24m Warning ProbeError pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Readiness probe error: Get "http://10.132.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 24m Warning Unhealthy pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Readiness probe failed: Get "http://10.132.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Normal ScalingReplicaSet deployment/e2e-inf-b-6cf6a0-openshift-default Scaled up replica set e2e-inf-b-6cf6a0-openshift-default-6db59cf67b from 0 to 1 24m Normal AddedInterface pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Add eth0 [10.132.0.43/23] from ovn-kubernetes 24m Normal AddedLabel gateway/e2e-inf-b-6cf6a0 Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-6cf6a0 24m Normal SuccessfulCreate replicaset/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b Created pod: e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr 23m Normal Pulled pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Started pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Started container istio-proxy 23m Normal Started pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Started container istio-proxy 23m Normal Started pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Started container istio-proxy 23m Normal Created pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Created container: istio-proxy 23m Normal Pulled pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Created pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Created container: istio-proxy 23m Normal Pulled pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Created pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Created container: istio-proxy 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-inf-b-6cf6a0 Scaled up replica set payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8 from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8 Created pod: payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8 Error creating: pods "payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-6cf6a0: serviceaccount "payload-processing-e2e-inf-b-6cf6a0" not found 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-inf-b-6cf6a0 Scaled up replica set payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc Created pod: payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr 23m Warning FailedCreate replicaset/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc Error creating: pods "payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-6cf6a0: serviceaccount "payload-processing-e2e-inf-b-6cf6a0" not found 23m Normal Created pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Created container: payload-processing 23m Normal AddedInterface pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Add eth0 [10.133.0.58/23] from ovn-kubernetes 23m Normal Pulled pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Normal AddedInterface pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Add eth0 [10.133.0.57/23] from ovn-kubernetes 23m Normal Pulled pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Normal Created pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Created container: payload-pre-processing 23m Normal Started pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Started container payload-pre-processing 23m Normal Started pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Started container payload-processing 23m Warning ProbeError pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 23m Warning Unhealthy pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Startup probe failed: Get "http://10.132.0.36:15021/healthz/ready": dial tcp 10.132.0.36:15021: connect: connection refused 23m Warning ProbeError pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Startup probe error: Get "http://10.132.0.37:15021/healthz/ready": dial tcp 10.132.0.37:15021: connect: connection refused... 23m Warning ProbeError pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Startup probe error: Get "http://10.132.0.36:15021/healthz/ready": dial tcp 10.132.0.36:15021: connect: connection refused... 23m Warning Unhealthy pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Startup probe failed: Get "http://10.132.0.37:15021/healthz/ready": dial tcp 10.132.0.37:15021: connect: connection refused 23m Normal Started pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Started container istio-proxy 23m Normal Created pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Created container: istio-proxy 23m Normal Pulled pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Killing pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Stopping container istio-proxy 23m Normal Killing pod/payload-processing-e2e-ait-b21bfa16-7ddc596d8f-lzlrr Stopping container payload-processing 23m Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-ait-b21bfa16 Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-ait-b21bfa16: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-ait-b21bfa16": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-ait-b21bfa16, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9157a918-541f-4f88-b85f-eb69d600786e, UID in object meta: 23m Warning Unhealthy pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Startup probe failed: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused 23m Warning ProbeError pod/e2e-ait-b21bfa16-openshift-default-65d6c97bc4-4kpkr Startup probe error: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused... 23m Normal Killing pod/payload-pre-processing-e2e-ait-b21bfa16-5849646f46-qfctz Stopping container payload-pre-processing 23m Normal AddedLabel gateway/e2e-migrate-fb988ee6-gw Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-fb988ee6-gw 23m Normal ScalingReplicaSet deployment/e2e-migrate-fb988ee6-gw-openshift-default Scaled up replica set e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d from 0 to 1 23m Normal SuccessfulCreate replicaset/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d Created pod: e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw 23m Normal AddedInterface pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Add eth0 [10.132.0.45/23] from ovn-kubernetes 23m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db Created pod: payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t 23m Normal ScalingReplicaSet deployment/payload-processing-e2e-migrate-fb988ee6 Scaled up replica set payload-processing-e2e-migrate-fb988ee6-dc9c5f958 from 0 to 1 23m Normal SuccessfulCreate replicaset/payload-processing-e2e-migrate-fb988ee6-dc9c5f958 Created pod: payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 23m Warning FailedCreate replicaset/payload-processing-e2e-migrate-fb988ee6-dc9c5f958 Error creating: pods "payload-processing-e2e-migrate-fb988ee6-dc9c5f958-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-fb988ee6: serviceaccount "payload-processing-e2e-migrate-fb988ee6" not found 23m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-migrate-fb988ee6 Scaled up replica set payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db from 0 to 1 23m Warning FailedCreate replicaset/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db Error creating: pods "payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-fb988ee6: serviceaccount "payload-processing-e2e-migrate-fb988ee6" not found 23m Normal Started pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Started container istio-proxy 23m Normal Pulled pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Normal Started pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Started container payload-processing 23m Normal Created pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Created container: payload-processing 23m Normal AddedInterface pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Add eth0 [10.132.0.46/23] from ovn-kubernetes 23m Normal Created pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Created container: istio-proxy 23m Normal Pulled pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal AddedInterface pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Add eth0 [10.133.0.59/23] from ovn-kubernetes 23m Normal Started pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Started container payload-pre-processing 23m Normal Created pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Created container: payload-pre-processing 23m Normal Pulled pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 23m Warning Unhealthy pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Startup probe failed: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused 23m Warning ProbeError pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Startup probe error: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused... 23m Normal Started pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Started container istio-proxy 23m Normal Pulled pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 23m Normal Created pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Created container: istio-proxy 23m Warning Unhealthy pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused 23m Warning ProbeError pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Startup probe error: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused... 22m Normal Killing pod/payload-processing-e2e-migrate-fb988ee6-dc9c5f958-trqn2 Stopping container payload-processing 22m Normal Killing pod/payload-pre-processing-e2e-migrate-fb988ee6-6fc9c645db-gj64t Stopping container payload-pre-processing 22m Normal Killing pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Stopping container istio-proxy 22m Warning ProbeError pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Readiness probe error: HTTP probe failed with statuscode: 503... 22m Normal ScalingReplicaSet deployment/e2e-ait-963aea34-openshift-default Scaled up replica set e2e-ait-963aea34-openshift-default-6db4564b96 from 0 to 1 22m Warning Unhealthy pod/e2e-migrate-fb988ee6-gw-openshift-default-684f6b4d4d-rh6tw Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal SuccessfulCreate replicaset/e2e-ait-963aea34-openshift-default-6db4564b96 Created pod: e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 22m Normal AddedLabel gateway/e2e-ait-963aea34 Added label istio.io/rev=openshift-gateway to gateway e2e-ait-963aea34 22m Normal AddedInterface pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Add eth0 [10.132.0.48/23] from ovn-kubernetes 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ait-963aea34-578447dc94 Created pod: payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ait-963aea34 Scaled up replica set payload-pre-processing-e2e-ait-963aea34-578447dc94 from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-ait-963aea34-578447dc94 Error creating: pods "payload-pre-processing-e2e-ait-963aea34-578447dc94-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-963aea34: serviceaccount "payload-processing-e2e-ait-963aea34" not found 22m Warning FailedCreate replicaset/payload-processing-e2e-ait-963aea34-5df966b875 Error creating: pods "payload-processing-e2e-ait-963aea34-5df966b875-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-963aea34: serviceaccount "payload-processing-e2e-ait-963aea34" not found 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-ait-963aea34-5df966b875 Created pod: payload-processing-e2e-ait-963aea34-5df966b875-fxngj 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-ait-963aea34 Scaled up replica set payload-processing-e2e-ait-963aea34-5df966b875 from 0 to 1 22m Normal Created pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Created container: payload-pre-processing 22m Normal Pulled pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal AddedInterface pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Add eth0 [10.132.0.49/23] from ovn-kubernetes 22m Normal Pulled pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal Started pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Started container payload-pre-processing 22m Normal Started pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Started container payload-processing 22m Normal Created pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Created container: payload-processing 22m Normal AddedInterface pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Add eth0 [10.132.0.50/23] from ovn-kubernetes 22m Normal Started pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Started container istio-proxy 22m Normal Pulled pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 22m Normal Created pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Created container: istio-proxy 22m Warning Unhealthy pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused 22m Warning ProbeError pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused... 22m Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-ait-963aea34 Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-ait-963aea34: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-ait-963aea34": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-ait-963aea34, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 600ae185-88f8-45fb-8608-6c6decd86e12, UID in object meta: 22m Normal Killing pod/payload-pre-processing-e2e-ait-963aea34-578447dc94-q4bmw Stopping container payload-pre-processing 22m Normal Killing pod/payload-processing-e2e-ait-963aea34-5df966b875-fxngj Stopping container payload-processing 22m Normal Killing pod/e2e-ait-963aea34-openshift-default-6db4564b96-nvfr9 Stopping container istio-proxy 22m Normal ScalingReplicaSet deployment/e2e-derive-219b8581-openshift-default Scaled up replica set e2e-derive-219b8581-openshift-default-c556c6d48 from 0 to 1 22m Normal AddedLabel gateway/e2e-derive-219b8581 Added label istio.io/rev=openshift-gateway to gateway e2e-derive-219b8581 22m Normal SuccessfulCreate replicaset/e2e-derive-219b8581-openshift-default-c556c6d48 Created pod: e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg 22m Normal AddedInterface pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Add eth0 [10.132.0.52/23] from ovn-kubernetes 22m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-derive-219b8581-579cffd574 Created pod: payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k 22m Warning FailedCreate replicaset/payload-processing-e2e-derive-219b8581-7bb49b54f7 Error creating: pods "payload-processing-e2e-derive-219b8581-7bb49b54f7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-219b8581: serviceaccount "payload-processing-e2e-derive-219b8581" not found 22m Normal SuccessfulCreate replicaset/payload-processing-e2e-derive-219b8581-7bb49b54f7 Created pod: payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr 22m Normal ScalingReplicaSet deployment/payload-processing-e2e-derive-219b8581 Scaled up replica set payload-processing-e2e-derive-219b8581-7bb49b54f7 from 0 to 1 22m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-derive-219b8581 Scaled up replica set payload-pre-processing-e2e-derive-219b8581-579cffd574 from 0 to 1 22m Warning FailedCreate replicaset/payload-pre-processing-e2e-derive-219b8581-579cffd574 Error creating: pods "payload-pre-processing-e2e-derive-219b8581-579cffd574-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-219b8581: serviceaccount "payload-processing-e2e-derive-219b8581" not found 22m Normal Created pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Created container: payload-processing 22m Normal AddedInterface pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Add eth0 [10.132.0.53/23] from ovn-kubernetes 22m Normal AddedInterface pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Add eth0 [10.132.0.54/23] from ovn-kubernetes 22m Normal Started pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Started container payload-pre-processing 22m Normal Pulled pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal Created pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Created container: payload-pre-processing 22m Normal Pulled pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 22m Normal Started pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Started container payload-processing 21m Warning Unhealthy pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Readiness probe failed: Get "http://10.132.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 21m Warning ProbeError pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Readiness probe error: Get "http://10.132.0.52:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 21m Normal Killing pod/payload-processing-e2e-derive-219b8581-7bb49b54f7-5hhzr Stopping container payload-processing 21m Normal Killing pod/payload-pre-processing-e2e-derive-219b8581-579cffd574-smg2k Stopping container payload-pre-processing 21m Normal Created pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Created container: istio-proxy 21m Normal Started pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Started container istio-proxy 21m Normal Pulled pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 21m Normal Killing pod/e2e-derive-219b8581-openshift-default-c556c6d48-zpxhg Stopping container istio-proxy 19m Normal AddedLabel gateway/e2e-mt-c3f23ba4 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-c3f23ba4 19m Normal AddedInterface pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Add eth0 [10.132.0.57/23] from ovn-kubernetes 19m Normal ScalingReplicaSet deployment/e2e-mt-c3f23ba4-openshift-default Scaled up replica set e2e-mt-c3f23ba4-openshift-default-5cb4bcc884 from 0 to 1 19m Normal SuccessfulCreate replicaset/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884 Created pod: e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k 19m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4 Error creating: pods "payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c3f23ba4: serviceaccount "payload-processing-e2e-mt-c3f23ba4" not found 19m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f Created pod: payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn 19m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-c3f23ba4 Scaled up replica set payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f from 0 to 1 19m Warning FailedCreate replicaset/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f Error creating: pods "payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-c3f23ba4: serviceaccount "payload-processing-e2e-mt-c3f23ba4" not found 19m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-c3f23ba4 Scaled up replica set payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4 from 0 to 1 19m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4 Created pod: payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb 19m Normal Created pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Created container: payload-processing 19m Normal Pulled pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal Started pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Started container payload-processing 19m Normal AddedInterface pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Add eth0 [10.132.0.59/23] from ovn-kubernetes 19m Normal AddedInterface pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Add eth0 [10.132.0.58/23] from ovn-kubernetes 19m Normal Pulled pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 19m Normal Created pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Created container: payload-pre-processing 19m Normal Started pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Started container payload-pre-processing 19m Warning Unhealthy pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 19m Warning ProbeError pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 19m Normal Created pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Created container: istio-proxy 19m Normal Pulled pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 19m Normal Started pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Started container istio-proxy 19m Warning Unhealthy pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused 19m Warning ProbeError pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused... 19m Normal Killing pod/payload-processing-e2e-mt-c3f23ba4-6b6bccbc8f-6lzjn Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-mt-c3f23ba4-7cf6f57ff4-vndzb Stopping container payload-pre-processing 19m Normal Killing pod/payload-processing-e2e-inf-a-f91f97-597bb7c965-7nr47 Stopping container payload-processing 19m Normal Killing pod/payload-pre-processing-e2e-inf-a-f91f97-7b55974cc4-2xnxb Stopping container payload-pre-processing 19m Normal Killing pod/e2e-mt-c3f23ba4-openshift-default-5cb4bcc884-rlv6k Stopping container istio-proxy 18m Normal Killing pod/e2e-inf-a-f91f97-openshift-default-7dc8555bfd-j5pk8 Stopping container istio-proxy 18m Normal Killing pod/payload-pre-processing-e2e-inf-b-6cf6a0-68d7c46bc8-6gqnr Stopping container payload-pre-processing 18m Normal Killing pod/payload-processing-e2e-inf-b-6cf6a0-7c9ff4fcdc-4qfcr Stopping container payload-processing 18m Normal AddedInterface pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Add eth0 [10.132.0.63/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8 Created pod: e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w1-9ac6b3-openshift-default Scaled up replica set e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8 from 0 to 1 18m Normal AddedLabel gateway/e2e-shared-a-w1-9ac6b3 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w1-9ac6b3 18m Normal Started pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Started container payload-processing 18m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc Error creating: pods "payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-9ac6b3: serviceaccount "payload-processing-e2e-shared-a-w1-9ac6b3" not found 18m Normal AddedInterface pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Add eth0 [10.133.0.64/23] from ovn-kubernetes 18m Normal Created pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Created container: payload-processing 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w1-9ac6b3 Scaled up replica set payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8 Created pod: payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8 Error creating: pods "payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w1-9ac6b3: serviceaccount "payload-processing-e2e-shared-a-w1-9ac6b3" not found 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w1-9ac6b3 Scaled up replica set payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc Created pod: payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz 18m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Add eth0 [10.133.0.63/23] from ovn-kubernetes 18m Normal Pulled pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Started pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Started container payload-pre-processing 18m Normal Created pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Created container: payload-pre-processing 18m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Killing pod/e2e-inf-b-6cf6a0-openshift-default-6db59cf67b-2scsr Stopping container istio-proxy 18m Normal AddedLabel gateway/e2e-shared-a-w2-72e262 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-w2-72e262 18m Normal AddedInterface pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Add eth0 [10.132.0.64/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/e2e-shared-a-w2-72e262-openshift-default-697f564d74 Created pod: e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt 18m Normal ScalingReplicaSet deployment/e2e-shared-a-w2-72e262-openshift-default Scaled up replica set e2e-shared-a-w2-72e262-openshift-default-697f564d74 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594 Error creating: pods "payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-72e262: serviceaccount "payload-processing-e2e-shared-a-w2-72e262" not found 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-a-w2-72e262 Scaled up replica set payload-processing-e2e-shared-a-w2-72e262-794875d7b5 from 0 to 1 18m Warning FailedCreate replicaset/payload-processing-e2e-shared-a-w2-72e262-794875d7b5 Error creating: pods "payload-processing-e2e-shared-a-w2-72e262-794875d7b5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-w2-72e262: serviceaccount "payload-processing-e2e-shared-a-w2-72e262" not found 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-a-w2-72e262-794875d7b5 Created pod: payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-a-w2-72e262 Scaled up replica set payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594 Created pod: payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 18m Normal Created pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Created container: payload-processing 18m Normal AddedInterface pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Add eth0 [10.132.0.65/23] from ovn-kubernetes 18m Normal Started pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Started container payload-pre-processing 18m Normal Created pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Created container: payload-pre-processing 18m Normal AddedInterface pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Add eth0 [10.132.0.66/23] from ovn-kubernetes 18m Normal Pulled pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Started pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Started container payload-processing 18m Normal Pulled pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Pulled pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 18m Normal Started pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Started container istio-proxy 18m Normal Created pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Created container: istio-proxy 18m Normal SuccessfulCreate replicaset/e2e-shared-b-w1-266c89-openshift-default-58d56d4678 Created pod: e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j 18m Warning Unhealthy pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused 18m Normal AddedLabel gateway/e2e-shared-b-w1-266c89 Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w1-266c89 18m Warning ProbeError pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused... 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w1-266c89-openshift-default Scaled up replica set e2e-shared-b-w1-266c89-openshift-default-58d56d4678 from 0 to 1 18m Normal AddedInterface pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Add eth0 [10.132.0.67/23] from ovn-kubernetes 18m Normal AddedLabel gateway/e2e-shared-b-w2-99b43e Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-w2-99b43e 18m Normal ScalingReplicaSet deployment/e2e-shared-b-w2-99b43e-openshift-default Scaled up replica set e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b from 0 to 1 18m Normal SuccessfulCreate replicaset/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b Created pod: e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 18m Normal AddedInterface pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Add eth0 [10.133.0.65/23] from ovn-kubernetes 18m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w1-266c89-7b444bc549 Error creating: pods "payload-processing-e2e-shared-b-w1-266c89-7b444bc549-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-266c89: serviceaccount "payload-processing-e2e-shared-b-w1-266c89" not found 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978 Error creating: pods "payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w1-266c89: serviceaccount "payload-processing-e2e-shared-b-w1-266c89" not found 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978 Created pod: payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w1-266c89 Scaled up replica set payload-pre-processing-e2e-shared-b-w1-266c89-d99785978 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w1-266c89-7b444bc549 Created pod: payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w1-266c89 Scaled up replica set payload-processing-e2e-shared-b-w1-266c89-7b444bc549 from 0 to 1 18m Warning FailedCreate replicaset/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d Error creating: pods "payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e: serviceaccount "payload-processing-e2e-shared-b-w2-99b43e" not found 18m Warning FailedCreate replicaset/payload-processing-e2e-shared-b-w2-99b43e-76dccf875 Error creating: pods "payload-processing-e2e-shared-b-w2-99b43e-76dccf875-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-w2-99b43e: serviceaccount "payload-processing-e2e-shared-b-w2-99b43e" not found 18m Normal Pulled pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Created pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Created container: payload-processing 18m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Add eth0 [10.132.0.68/23] from ovn-kubernetes 18m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d Created pod: payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb 18m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-shared-b-w2-99b43e Scaled up replica set payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d from 0 to 1 18m Normal Started pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Started container payload-processing 18m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Created pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Created container: payload-pre-processing 18m Normal Started pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Started container payload-pre-processing 18m Normal ScalingReplicaSet deployment/payload-processing-e2e-shared-b-w2-99b43e Scaled up replica set payload-processing-e2e-shared-b-w2-99b43e-76dccf875 from 0 to 1 18m Normal SuccessfulCreate replicaset/payload-processing-e2e-shared-b-w2-99b43e-76dccf875 Created pod: payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh 18m Normal AddedInterface pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Add eth0 [10.133.0.67/23] from ovn-kubernetes 18m Normal Started pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Started container payload-processing 18m Normal Created pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Created container: payload-processing 18m Normal Pulled pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal AddedInterface pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Add eth0 [10.132.0.70/23] from ovn-kubernetes 18m Normal Created pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Created container: payload-pre-processing 18m Normal AddedInterface pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Add eth0 [10.132.0.69/23] from ovn-kubernetes 18m Normal Pulled pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 18m Normal Started pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Started container payload-pre-processing 18m Normal Started pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Started container istio-proxy 18m Normal Created pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Created container: istio-proxy 18m Normal Pulled pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 17m Warning ProbeError pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Startup probe error: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused... 17m Warning Unhealthy pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Startup probe failed: Get "http://10.132.0.64:15021/healthz/ready": dial tcp 10.132.0.64:15021: connect: connection refused 17m Normal Started pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Started container istio-proxy 17m Normal Created pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Created container: istio-proxy 17m Normal Pulled pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 17m Normal Created pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Created container: istio-proxy 17m Normal Pulled pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 17m Normal Started pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Started container istio-proxy 17m Warning ProbeError pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 17m Warning Unhealthy pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 17m Warning Unhealthy pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Startup probe failed: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused 17m Warning ProbeError pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Startup probe error: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused... 17m Normal AddedInterface pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Add eth0 [10.132.0.71/23] from ovn-kubernetes 17m Normal SuccessfulCreate replicaset/e2e-mt-e6f44882-openshift-default-69467ff6b8 Created pod: e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk 17m Normal ScalingReplicaSet deployment/e2e-mt-e6f44882-openshift-default Scaled up replica set e2e-mt-e6f44882-openshift-default-69467ff6b8 from 0 to 1 17m Normal AddedLabel gateway/e2e-mt-e6f44882 Added label istio.io/rev=openshift-gateway to gateway e2e-mt-e6f44882 17m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d Created pod: payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz 17m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-mt-e6f44882 Scaled up replica set payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d from 0 to 1 17m Normal Started pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Started container payload-pre-processing 17m Normal Created pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Created container: payload-pre-processing 17m Normal Pulled pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 17m Normal AddedInterface pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Add eth0 [10.133.0.68/23] from ovn-kubernetes 17m Normal AddedInterface pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Add eth0 [10.133.0.69/23] from ovn-kubernetes 17m Warning FailedCreate replicaset/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d Error creating: pods "payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-e6f44882: serviceaccount "payload-processing-e2e-mt-e6f44882" not found 17m Normal Pulled pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 17m Normal SuccessfulCreate replicaset/payload-processing-e2e-mt-e6f44882-b49b6749c Created pod: payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl 17m Normal ScalingReplicaSet deployment/payload-processing-e2e-mt-e6f44882 Scaled up replica set payload-processing-e2e-mt-e6f44882-b49b6749c from 0 to 1 17m Warning FailedCreate replicaset/payload-processing-e2e-mt-e6f44882-b49b6749c Error creating: pods "payload-processing-e2e-mt-e6f44882-b49b6749c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-e6f44882: serviceaccount "payload-processing-e2e-mt-e6f44882" not found 17m Normal Started pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Started container payload-processing 17m Normal Created pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Created container: payload-processing 17m Warning ProbeError pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Readiness probe error: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 17m Warning Unhealthy pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Readiness probe failed: Get "http://10.132.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 16m Normal Created pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Created container: istio-proxy 16m Normal Pulled pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 16m Normal Started pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Started container istio-proxy 16m Warning ProbeError pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 16m Warning Unhealthy pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 16m Normal Killing pod/payload-pre-processing-e2e-worker-w3-580e01-84f778859d-dqwdk Stopping container payload-pre-processing 16m Normal Killing pod/payload-processing-e2e-worker-w3-580e01-6885674c9-cfrxv Stopping container payload-processing 16m Normal Killing pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Stopping container istio-proxy 16m Warning Unhealthy pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Readiness probe failed: HTTP probe failed with statuscode: 503 16m Warning ProbeError pod/e2e-worker-w3-580e01-openshift-default-db458d66d-xt2xt Readiness probe error: HTTP probe failed with statuscode: 503... 15m Normal Killing pod/payload-processing-e2e-mt-e6f44882-b49b6749c-5c5nl Stopping container payload-processing 15m Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-mt-e6f44882 Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-mt-e6f44882: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-mt-e6f44882": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-mt-e6f44882, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 000242a4-efed-4ad3-9ed8-5914bcf11d71, UID in object meta: 15m Normal Killing pod/payload-pre-processing-e2e-mt-e6f44882-bb8bb6b5d-hjsxz Stopping container payload-pre-processing 15m Normal Killing pod/payload-processing-e2e-worker-w0-cc1b19-d96b479d9-xpzmd Stopping container payload-processing 15m Normal Killing pod/payload-pre-processing-e2e-worker-w0-cc1b19-7cd89f954b-46qxq Stopping container payload-pre-processing 15m Normal Killing pod/e2e-worker-w0-cc1b19-openshift-default-5cc7cfc5d-xg6vb Stopping container istio-proxy 14m Normal Killing pod/e2e-mt-e6f44882-openshift-default-69467ff6b8-mjlqk Stopping container istio-proxy 13m Normal AddedInterface pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Add eth0 [10.133.0.72/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/e2e-api-a-45f425-openshift-default-bbcc9f4d4 Created pod: e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz 13m Normal ScalingReplicaSet deployment/e2e-api-a-45f425-openshift-default Scaled up replica set e2e-api-a-45f425-openshift-default-bbcc9f4d4 from 0 to 1 13m Normal AddedLabel gateway/e2e-api-a-45f425 Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-45f425 13m Normal Pulled pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-a-45f425 Scaled up replica set payload-processing-e2e-api-a-45f425-685cfff579 from 0 to 1 13m Normal Started pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Started container payload-pre-processing 13m Normal AddedInterface pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Add eth0 [10.133.0.73/23] from ovn-kubernetes 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-a-45f425-d699cbf98 Error creating: pods "payload-pre-processing-e2e-api-a-45f425-d699cbf98-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-45f425: serviceaccount "payload-processing-e2e-api-a-45f425" not found 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-a-45f425 Scaled up replica set payload-pre-processing-e2e-api-a-45f425-d699cbf98 from 0 to 1 13m Normal AddedInterface pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Add eth0 [10.132.0.75/23] from ovn-kubernetes 13m Normal Pulled pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 13m Normal Created pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Created container: payload-processing 13m Normal Created pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Created container: payload-pre-processing 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-a-45f425-685cfff579 Created pod: payload-processing-e2e-api-a-45f425-685cfff579-szmt4 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-a-45f425-d699cbf98 Created pod: payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx 13m Warning FailedCreate replicaset/payload-processing-e2e-api-a-45f425-685cfff579 Error creating: pods "payload-processing-e2e-api-a-45f425-685cfff579-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-45f425: serviceaccount "payload-processing-e2e-api-a-45f425" not found 13m Normal Started pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Started container payload-processing 13m Normal Started pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Started container istio-proxy 13m Normal Created pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Created container: istio-proxy 13m Normal Pulled pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 13m Warning Unhealthy pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused 13m Warning ProbeError pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused... 13m Normal ScalingReplicaSet deployment/e2e-api-b-a1c584-openshift-default Scaled up replica set e2e-api-b-a1c584-openshift-default-847dcdd99d from 0 to 1 13m Normal SuccessfulCreate replicaset/e2e-api-b-a1c584-openshift-default-847dcdd99d Created pod: e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq 13m Normal AddedLabel gateway/e2e-api-b-a1c584 Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-a1c584 13m Normal AddedInterface pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Add eth0 [10.132.0.76/23] from ovn-kubernetes 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4 Error creating: pods "payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-a1c584: serviceaccount "payload-processing-e2e-api-b-a1c584" not found 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-api-b-a1c584 Scaled up replica set payload-processing-e2e-api-b-a1c584-7f67577f49 from 0 to 1 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-api-b-a1c584 Scaled up replica set payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4 from 0 to 1 12m Warning FailedCreate replicaset/payload-processing-e2e-api-b-a1c584-7f67577f49 Error creating: pods "payload-processing-e2e-api-b-a1c584-7f67577f49-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-a1c584: serviceaccount "payload-processing-e2e-api-b-a1c584" not found 12m Normal Started pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Started container payload-processing 12m Normal Pulled pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 12m Normal AddedInterface pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Add eth0 [10.133.0.74/23] from ovn-kubernetes 12m Normal Started pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Started container payload-pre-processing 12m Normal Created pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Created container: payload-pre-processing 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-api-b-a1c584-7f67577f49 Created pod: payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 12m Normal Created pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Created container: payload-processing 12m Normal Pulled pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 12m Normal AddedInterface pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Add eth0 [10.133.0.75/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4 Created pod: payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp 12m Normal Pulled pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 12m Normal Started pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Started container istio-proxy 12m Normal Created pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Created container: istio-proxy 12m Warning ProbeError pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 12m Warning Unhealthy pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 12m Normal Killing pod/payload-pre-processing-e2e-api-a-45f425-d699cbf98-lrjvx Stopping container payload-pre-processing 12m Normal Killing pod/payload-processing-e2e-api-a-45f425-685cfff579-szmt4 Stopping container payload-processing 12m Normal Killing pod/e2e-api-a-45f425-openshift-default-bbcc9f4d4-6nsnz Stopping container istio-proxy 11m Normal Killing pod/payload-pre-processing-e2e-api-b-a1c584-5bf4995bc4-vprbp Stopping container payload-pre-processing 11m Normal Killing pod/payload-processing-e2e-api-b-a1c584-7f67577f49-82qd5 Stopping container payload-processing 11m Normal Killing pod/e2e-api-b-a1c584-openshift-default-847dcdd99d-2nzxq Stopping container istio-proxy 11m Normal Killing pod/payload-processing-e2e-shared-a-w1-9ac6b3-64bdb7b8dc-n8mwz Stopping container payload-processing 11m Normal Killing pod/payload-pre-processing-e2e-shared-a-w1-9ac6b3-6f689fbdc8-nd695 Stopping container payload-pre-processing 11m Normal Killing pod/e2e-shared-a-w1-9ac6b3-openshift-default-7f864ccbd8-g2j5d Stopping container istio-proxy 11m Normal Killing pod/payload-processing-e2e-shared-b-w1-266c89-7b444bc549-x5xmk Stopping container payload-processing 11m Normal Killing pod/payload-pre-processing-e2e-shared-b-w1-266c89-d99785978-zz7tl Stopping container payload-pre-processing 11m Normal Killing pod/e2e-shared-b-w1-266c89-openshift-default-58d56d4678-wc78j Stopping container istio-proxy 10m Normal AddedLabel gateway/e2e-ipp-a-bd3aad Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-bd3aad 10m Normal SuccessfulCreate replicaset/e2e-ipp-a-bd3aad-openshift-default-66977bc66d Created pod: e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx 10m Normal ScalingReplicaSet deployment/e2e-ipp-a-bd3aad-openshift-default Scaled up replica set e2e-ipp-a-bd3aad-openshift-default-66977bc66d from 0 to 1 10m Normal AddedInterface pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Add eth0 [10.133.0.77/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4 Created pod: payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-a-bd3aad Scaled up replica set payload-processing-e2e-ipp-a-bd3aad-5c658ccc9 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-a-bd3aad Scaled up replica set payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4 from 0 to 1 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4 Error creating: pods "payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bd3aad: serviceaccount "payload-processing-e2e-ipp-a-bd3aad" not found 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9 Created pod: payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg 10m Warning FailedCreate replicaset/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9 Error creating: pods "payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-bd3aad: serviceaccount "payload-processing-e2e-ipp-a-bd3aad" not found 10m Normal Created pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Created container: payload-pre-processing 10m Normal Pulled pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Add eth0 [10.133.0.79/23] from ovn-kubernetes 10m Normal Started pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Started container payload-processing 10m Normal Pulled pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal Created pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Created container: payload-processing 10m Normal Started pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Started container payload-pre-processing 10m Normal AddedInterface pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Add eth0 [10.133.0.80/23] from ovn-kubernetes 10m Warning ProbeError pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Readiness probe error: Get "http://10.133.0.77:15021/healthz/ready": read tcp 10.133.0.2:37844->10.133.0.77:15021: read: connection reset by peer... 10m Warning Unhealthy pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Readiness probe failed: Get "http://10.133.0.77:15021/healthz/ready": read tcp 10.133.0.2:37844->10.133.0.77:15021: read: connection reset by peer 10m Normal Started pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Started container istio-proxy 10m Normal Created pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Created container: istio-proxy 10m Normal Pulled pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 10m Warning Unhealthy pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Startup probe failed: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused 10m Warning ProbeError pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Startup probe error: Get "http://10.133.0.77:15021/healthz/ready": dial tcp 10.133.0.77:15021: connect: connection refused... 10m Normal SuccessfulCreate replicaset/e2e-ipp-b-f210d9-openshift-default-698757f6bd Created pod: e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq 10m Normal ScalingReplicaSet deployment/e2e-ipp-b-f210d9-openshift-default Scaled up replica set e2e-ipp-b-f210d9-openshift-default-698757f6bd from 0 to 1 10m Normal AddedLabel gateway/e2e-ipp-b-f210d9 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-f210d9 10m Normal AddedInterface pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Add eth0 [10.133.0.81/23] from ovn-kubernetes 10m Normal AddedInterface pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Add eth0 [10.132.0.86/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-b-f210d9-7df74697b7 Created pod: payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d Created pod: payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d Error creating: pods "payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f210d9: serviceaccount "payload-processing-e2e-ipp-b-f210d9" not found 10m Normal Started pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Started container payload-pre-processing 10m Normal Pulled pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-b-f210d9 Scaled up replica set payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d from 0 to 1 10m Warning FailedCreate replicaset/payload-processing-e2e-ipp-b-f210d9-7df74697b7 Error creating: pods "payload-processing-e2e-ipp-b-f210d9-7df74697b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-f210d9: serviceaccount "payload-processing-e2e-ipp-b-f210d9" not found 10m Normal Created pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Created container: payload-pre-processing 10m Normal Pulled pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 10m Normal AddedInterface pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Add eth0 [10.132.0.85/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-b-f210d9 Scaled up replica set payload-processing-e2e-ipp-b-f210d9-7df74697b7 from 0 to 1 10m Normal Started pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Started container payload-processing 10m Normal Created pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Created container: payload-processing 9m52s Warning ProbeError pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Readiness probe error: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 9m52s Warning Unhealthy pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Readiness probe failed: Get "http://10.133.0.81:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 9m50s Normal Created pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Created container: istio-proxy 9m50s Normal Started pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Started container istio-proxy 9m50s Normal Pulled pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 9m45s Warning ProbeError pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Startup probe error: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused... 9m45s Warning Unhealthy pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Startup probe failed: Get "http://10.133.0.81:15021/healthz/ready": dial tcp 10.133.0.81:15021: connect: connection refused 9m9s Normal SuccessfulCreate replicaset/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb Created pod: e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g 9m9s Normal AddedLabel gateway/e2e-ipp-cleanup-953b43 Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-cleanup-953b43 9m9s Normal ScalingReplicaSet deployment/e2e-ipp-cleanup-953b43-openshift-default Scaled up replica set e2e-ipp-cleanup-953b43-openshift-default-68f67744cb from 0 to 1 9m8s Normal AddedInterface pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Add eth0 [10.133.0.82/23] from ovn-kubernetes 9m2s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-ipp-cleanup-953b43 Scaled up replica set payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d from 0 to 1 9m2s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d Created pod: payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj 9m2s Warning FailedCreate replicaset/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d Error creating: pods "payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-953b43: serviceaccount "payload-processing-e2e-ipp-cleanup-953b43" not found 9m2s Normal SuccessfulCreate replicaset/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c Created pod: payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 9m2s Normal ScalingReplicaSet deployment/payload-processing-e2e-ipp-cleanup-953b43 Scaled up replica set payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c from 0 to 1 9m2s Warning FailedCreate replicaset/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c Error creating: pods "payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-953b43: serviceaccount "payload-processing-e2e-ipp-cleanup-953b43" not found 9m1s Normal Pulled pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 9m1s Normal AddedInterface pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Add eth0 [10.133.0.83/23] from ovn-kubernetes 9m1s Normal Started pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Started container payload-pre-processing 9m1s Normal AddedInterface pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Add eth0 [10.132.0.87/23] from ovn-kubernetes 9m1s Normal Pulled pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 9m1s Normal Started pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Started container payload-processing 9m1s Normal Created pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Created container: payload-processing 9m1s Normal Created pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Created container: payload-pre-processing 8m45s Normal Pulled pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 8m45s Normal Created pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Created container: istio-proxy 8m45s Normal Started pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Started container istio-proxy 8m39s Warning Unhealthy pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 8m39s Warning ProbeError pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 8m33s Normal Killing pod/payload-processing-e2e-ipp-cleanup-953b43-865b99dc8c-4lcb2 Stopping container payload-processing 8m33s Normal Killing pod/payload-pre-processing-e2e-ipp-cleanup-953b43-6458fdff9d-b9sfj Stopping container payload-pre-processing 8m27s Normal Killing pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Stopping container istio-proxy 8m23s Warning Unhealthy pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Readiness probe failed: HTTP probe failed with statuscode: 503 8m23s Warning ProbeError pod/e2e-ipp-cleanup-953b43-openshift-default-68f67744cb-48m8g Readiness probe error: HTTP probe failed with statuscode: 503... 8m8s Normal Killing pod/payload-pre-processing-e2e-ipp-a-bd3aad-566fd59dc4-bcpb2 Stopping container payload-pre-processing 8m8s Normal Killing pod/payload-processing-e2e-ipp-a-bd3aad-5c658ccc9-h5wpg Stopping container payload-processing 7m19s Normal Killing pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Stopping container istio-proxy 7m15s Warning ProbeError pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Readiness probe error: HTTP probe failed with statuscode: 503... 7m15s Warning Unhealthy pod/e2e-ipp-a-bd3aad-openshift-default-66977bc66d-g9qzx Readiness probe failed: HTTP probe failed with statuscode: 503 7m9s Warning FailedToUpdateEndpoint endpoints/payload-pre-processing-e2e-ipp-b-f210d9 Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-ipp-b-f210d9": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-ipp-b-f210d9, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0f4e67cf-3ca7-4a8b-8033-a8757e0bcd94, UID in object meta: 7m9s Normal Killing pod/payload-processing-e2e-ipp-b-f210d9-7df74697b7-8m67j Stopping container payload-processing 7m9s Normal Killing pod/payload-pre-processing-e2e-ipp-b-f210d9-6fdd6c549d-wlprn Stopping container payload-pre-processing 7m2s Normal Killing pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Stopping container istio-proxy 6m59s Warning Unhealthy pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Readiness probe failed: HTTP probe failed with statuscode: 503 6m59s Warning ProbeError pod/e2e-ipp-b-f210d9-openshift-default-698757f6bd-lp2sq Readiness probe error: HTTP probe failed with statuscode: 503... 6m51s Normal Killing pod/payload-processing-e2e-shared-a-w2-72e262-794875d7b5-nqmtb Stopping container payload-processing 6m51s Normal Killing pod/payload-pre-processing-e2e-shared-a-w2-72e262-75b57bb594-5nl28 Stopping container payload-pre-processing 6m45s Normal Killing pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Stopping container istio-proxy 6m42s Warning ProbeError pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Readiness probe error: HTTP probe failed with statuscode: 503... 6m42s Warning Unhealthy pod/e2e-shared-a-w2-72e262-openshift-default-697f564d74-rzjqt Readiness probe failed: HTTP probe failed with statuscode: 503 6m34s Normal Killing pod/payload-processing-e2e-shared-b-w2-99b43e-76dccf875-6bqqh Stopping container payload-processing 6m34s Normal Killing pod/payload-pre-processing-e2e-shared-b-w2-99b43e-5dd5c4559d-nzmpb Stopping container payload-pre-processing 6m27s Normal Killing pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Stopping container istio-proxy 6m23s Warning ProbeError pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Readiness probe error: HTTP probe failed with statuscode: 503... 6m23s Warning Unhealthy pod/e2e-shared-b-w2-99b43e-openshift-default-85c9b5cc9b-8ztq8 Readiness probe failed: HTTP probe failed with statuscode: 503