Time Namespace Component RelatedObject Reason Message

openshift-ingress

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Scheduled

Successfully assigned openshift-ingress/e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t to ip-10-0-131-202.ec2.internal

openshift-ingress

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Scheduled

Successfully assigned openshift-ingress/e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4 to ip-10-0-131-202.ec2.internal

openshift-ingress

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tpkbb to ip-10-0-132-216.ec2.internal

openshift-ingress

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Scheduled

Successfully assigned openshift-ingress/e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8 to ip-10-0-131-202.ec2.internal

openshift-ingress

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Scheduled

Successfully assigned openshift-ingress/e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk to ip-10-0-131-202.ec2.internal

openshift-ingress

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Scheduled

Successfully assigned openshift-ingress/e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-processing-5cc6cd444b-rm4bg

Scheduled

Successfully assigned openshift-ingress/payload-processing-5cc6cd444b-rm4bg to ip-10-0-132-216.ec2.internal

openshift-ingress

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65 to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Scheduled

Successfully assigned openshift-ingress/e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2 to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Scheduled

Successfully assigned openshift-ingress/e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472 to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5 to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Scheduled

Successfully assigned openshift-ingress/e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr to ip-10-0-132-216.ec2.internal

openshift-ingress

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Scheduled

Successfully assigned openshift-ingress/e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89 to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48 to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Scheduled

Successfully assigned openshift-ingress/e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn to ip-10-0-131-202.ec2.internal

openshift-ingress

kube-auth-proxy-5fb4d8769-k7rn9

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-5fb4d8769-k7rn9 to ip-10-0-132-216.ec2.internal

openshift-ingress

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

Scheduled

Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh to ip-10-0-132-216.ec2.internal

openshift-ingress

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Scheduled

Successfully assigned openshift-ingress/e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-api-b-3adaab-8566476754-tv72x to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7 to ip-10-0-131-202.ec2.internal

openshift-ingress

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Scheduled

Successfully assigned openshift-ingress/e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-pre-processing-687dbb6cf6-wjddx

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-687dbb6cf6-wjddx to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Scheduled

Successfully assigned openshift-ingress/e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7 to ip-10-0-131-202.ec2.internal

openshift-ingress

router-default-6bfbd696bc-8jlrm

Scheduled

Successfully assigned openshift-ingress/router-default-6bfbd696bc-8jlrm to ip-10-0-132-216.ec2.internal

openshift-ingress

router-default-6bfbd696bc-8jlrm

FailedScheduling

0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

openshift-ingress

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674 to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4 to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c to ip-10-0-130-131.ec2.internal

openshift-ingress

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Scheduled

Successfully assigned openshift-ingress/e2e-api-b-3adaab-openshift-default-659664b78-l8skx to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-mt-60d3c925-96db7746-6trnp to ip-10-0-130-131.ec2.internal

openshift-ingress

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x to ip-10-0-131-202.ec2.internal

openshift-ingress

kube-auth-proxy-5fb4d8769-zxv9t

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-5fb4d8769-zxv9t to ip-10-0-130-131.ec2.internal

openshift-ingress

istiod-openshift-gateway-5677ccb49b-6hl27

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-6hl27 to ip-10-0-131-202.ec2.internal

openshift-ingress

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Scheduled

Successfully assigned openshift-ingress/e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt to ip-10-0-131-202.ec2.internal

openshift-ingress

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b to ip-10-0-131-202.ec2.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-6bfbd696bc from 0 to 1

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x12)

openshift-ingress

replicaset-controller

router-default-6bfbd696bc

FailedCreate

Error creating: pods "router-default-6bfbd696bc-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
(x16)

openshift-ingress

replicaset-controller

router-default-6bfbd696bc

FailedCreate

Error creating: pods "router-default-6bfbd696bc-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

openshift-ingress

replicaset-controller

router-default-6bfbd696bc

SuccessfulCreate

Created pod: router-default-6bfbd696bc-8jlrm
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts
(x6)

openshift-ingress

kubelet

router-default-6bfbd696bc-8jlrm

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
(x6)

openshift-ingress

kubelet

router-default-6bfbd696bc-8jlrm

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found

openshift-ingress

kubelet

router-default-6bfbd696bc-8jlrm

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559" already present on machine

openshift-ingress

kubelet

router-default-6bfbd696bc-8jlrm

Created

Created container: router

openshift-ingress

multus

router-default-6bfbd696bc-8jlrm

AddedInterface

Add eth0 [10.132.0.12/23] from ovn-kubernetes

openshift-ingress

kubelet

router-default-6bfbd696bc-8jlrm

Started

Started container router

openshift-ingress

deployment-controller

kube-auth-proxy

ScalingReplicaSet

Scaled up replica set kube-auth-proxy-5fb4d8769 from 0 to 2
(x5)

openshift-ingress

replicaset-controller

kube-auth-proxy-5fb4d8769

FailedCreate

Error creating: pods "kube-auth-proxy-5fb4d8769-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-zxv9t

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-k7rn9

FailedMount

MountVolume.SetUp failed for volume "tls-certs" : secret "kube-auth-proxy-tls" not found

openshift-ingress

multus

kube-auth-proxy-5fb4d8769-zxv9t

AddedInterface

Add eth0 [10.134.0.22/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

kube-auth-proxy-5fb4d8769

SuccessfulCreate

Created pod: kube-auth-proxy-5fb4d8769-k7rn9

openshift-ingress

replicaset-controller

kube-auth-proxy-5fb4d8769

SuccessfulCreate

Created pod: kube-auth-proxy-5fb4d8769-zxv9t

openshift-ingress

multus

kube-auth-proxy-5fb4d8769-k7rn9

AddedInterface

Add eth0 [10.132.0.33/23] from ovn-kubernetes

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-k7rn9

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1

openshift-ingress

replicaset-controller

istiod-openshift-gateway-5677ccb49b

SuccessfulCreate

Created pod: istiod-openshift-gateway-5677ccb49b-6hl27

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-zxv9t

Created

Created container: kube-auth-proxy

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-zxv9t

Started

Started container kube-auth-proxy

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"

openshift-ingress

multus

istiod-openshift-gateway-5677ccb49b-6hl27

AddedInterface

Add eth0 [10.133.0.24/23] from ovn-kubernetes

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-zxv9t

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.914s (2.914s including waiting). Image size: 265120132 bytes.

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-k7rn9

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 3.181s (3.181s including waiting). Image size: 265120132 bytes.

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-k7rn9

Created

Created container: kube-auth-proxy

openshift-ingress

kubelet

kube-auth-proxy-5fb4d8769-k7rn9

Started

Started container kube-auth-proxy

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 3.992s (3.992s including waiting). Image size: 181126104 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-6hl27

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

deployment-controller

data-science-gateway-data-science-gateway-class

ScalingReplicaSet

Scaled up replica set data-science-gateway-data-science-gateway-class-d66cd8c7c from 0 to 1

openshift-ingress

replicaset-controller

data-science-gateway-data-science-gateway-class-d66cd8c7c

SuccessfulCreate

Created pod: data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"

openshift-ingress

multus

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

AddedInterface

Add eth0 [10.132.0.34/23] from ovn-kubernetes

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

Started

Started container istio-proxy

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

Created

Created container: istio-proxy

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-d66cd8c7c-j65fh

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.459s (2.459s including waiting). Image size: 181318953 bytes.

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

gateway_labeler_controller

maas-default-gateway

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-78bb78ff78 from 0 to 1

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-78bb78ff78

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-78bb78ff78-tpkbb

openshift-ingress

multus

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

AddedInterface

Add eth0 [10.132.0.42/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-pre-processing

ScalingReplicaSet

Scaled up replica set payload-pre-processing-687dbb6cf6 from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-5cc6cd444b

SuccessfulCreate

Created pod: payload-processing-5cc6cd444b-rm4bg

openshift-ingress

deployment-controller

payload-processing

ScalingReplicaSet

Scaled up replica set payload-processing-5cc6cd444b from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-687dbb6cf6

SuccessfulCreate

Created pod: payload-pre-processing-687dbb6cf6-wjddx
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-687dbb6cf6

FailedCreate

Error creating: pods "payload-pre-processing-687dbb6cf6-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found
(x6)

openshift-ingress

replicaset-controller

payload-processing-5cc6cd444b

FailedCreate

Error creating: pods "payload-processing-5cc6cd444b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-wjddx

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"

openshift-ingress

multus

payload-processing-5cc6cd444b-rm4bg

AddedInterface

Add eth0 [10.132.0.46/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-5cc6cd444b-rm4bg

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"

openshift-ingress

multus

payload-pre-processing-687dbb6cf6-wjddx

AddedInterface

Add eth0 [10.133.0.35/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-wjddx

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.721s (1.721s including waiting). Image size: 165907129 bytes.

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-wjddx

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-wjddx

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-processing-5cc6cd444b-rm4bg

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-5cc6cd444b-rm4bg

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-5cc6cd444b-rm4bg

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.8s (2.8s including waiting). Image size: 165907129 bytes.
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

Started

Started container istio-proxy
(x4)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

Unhealthy

Startup probe failed: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-78bb78ff78-tpkbb

ProbeError

Startup probe error: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused body:

openshift-ingress

gateway_labeler_controller

e2e-ait-c5c29115

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ait-c5c29115

openshift-ingress

multus

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

AddedInterface

Add eth0 [10.133.0.42/23] from ovn-kubernetes

openshift-ingress

deployment-controller

e2e-ait-c5c29115-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ait-c5c29115-openshift-default-5b855d754 from 0 to 1

openshift-ingress

replicaset-controller

e2e-ait-c5c29115-openshift-default-5b855d754

SuccessfulCreate

Created pod: e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.587s (2.587s including waiting). Image size: 181318953 bytes.

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ait-c5c29115

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6 from 0 to 1
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-c5c29115: serviceaccount "payload-processing-e2e-ait-c5c29115" not found

openshift-ingress

multus

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

AddedInterface

Add eth0 [10.133.0.45/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

AddedInterface

Add eth0 [10.133.0.44/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-c5c29115-86f5fc4bb6

FailedCreate

Error creating: pods "payload-processing-e2e-ait-c5c29115-86f5fc4bb6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-c5c29115: serviceaccount "payload-processing-e2e-ait-c5c29115" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-c5c29115-86f5fc4bb6

SuccessfulCreate

Created pod: payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

openshift-ingress

deployment-controller

payload-processing-e2e-ait-c5c29115

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ait-c5c29115-86f5fc4bb6 from 0 to 1

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

Started

Started container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

Started

Started container payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Created

Created container: istio-proxy

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Started

Started container istio-proxy
(x4)

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Unhealthy

Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

ProbeError

Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-ait-c5c29115-86f5fc4bb6-qpn2x

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-c5c29115-6b4b6c56f6-7hlhk

Killing

Stopping container payload-pre-processing

openshift-ingress

gateway_labeler_controller

e2e-migrate-57a0d2c8-gw

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-57a0d2c8-gw

openshift-ingress

kubelet

e2e-ait-c5c29115-openshift-default-5b855d754-qhwxh

Killing

Stopping container istio-proxy

openshift-ingress

multus

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

AddedInterface

Add eth0 [10.133.0.46/23] from ovn-kubernetes

openshift-ingress

deployment-controller

e2e-migrate-57a0d2c8-gw-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888 from 0 to 1

openshift-ingress

replicaset-controller

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888

SuccessfulCreate

Created pod: e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-migrate-57a0d2c8-6d59987865

FailedCreate

Error creating: pods "payload-processing-e2e-migrate-57a0d2c8-6d59987865-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-57a0d2c8: serviceaccount "payload-processing-e2e-migrate-57a0d2c8" not found
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b

FailedCreate

Error creating: pods "payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-57a0d2c8: serviceaccount "payload-processing-e2e-migrate-57a0d2c8" not found

openshift-ingress

deployment-controller

payload-processing-e2e-migrate-57a0d2c8

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-migrate-57a0d2c8-6d59987865 from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-migrate-57a0d2c8-6d59987865

SuccessfulCreate

Created pod: payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

openshift-ingress

deployment-controller

payload-pre-processing-e2e-migrate-57a0d2c8

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b

SuccessfulCreate

Created pod: payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

openshift-ingress

multus

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

openshift-ingress

multus

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

Started

Started container payload-processing
(x2)

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

ProbeError

Readiness probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused body:

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Unhealthy

Readiness probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Started

Started container istio-proxy
(x4)

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

ProbeError

Startup probe error: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Unhealthy

Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-57a0d2c8-65d4dd4f9b-xzgf7

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-migrate-57a0d2c8-6d59987865-t4w7b

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Killing

Stopping container istio-proxy

openshift-ingress

gateway_labeler_controller

e2e-ait-ed80512f

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ait-ed80512f

openshift-ingress

deployment-controller

e2e-ait-ed80512f-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ait-ed80512f-openshift-default-7cbcf95486 from 0 to 1

openshift-ingress

replicaset-controller

e2e-ait-ed80512f-openshift-default-7cbcf95486

SuccessfulCreate

Created pod: e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

openshift-ingress

multus

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-processing-e2e-ait-ed80512f

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ait-ed80512f-b5fbc8894 from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-ed80512f-5f4d667548

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ait-ed80512f-5f4d667548-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-ed80512f: serviceaccount "payload-processing-e2e-ait-ed80512f" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-ed80512f-b5fbc8894

SuccessfulCreate

Created pod: payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x6)

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-ed80512f-b5fbc8894

FailedCreate

Error creating: pods "payload-processing-e2e-ait-ed80512f-b5fbc8894-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-ed80512f: serviceaccount "payload-processing-e2e-ait-ed80512f" not found

openshift-ingress

kubelet

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"

openshift-ingress

multus

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

AddedInterface

Add eth0 [10.134.0.35/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ait-ed80512f

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ait-ed80512f-5f4d667548 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-ed80512f-5f4d667548

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

openshift-ingress

kubelet

e2e-migrate-57a0d2c8-gw-openshift-default-594dd94888-gvxvt

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

AddedInterface

Add eth0 [10.133.0.52/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.501s (1.501s including waiting). Image size: 165907129 bytes.

openshift-ingress

kubelet

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Started

Started container payload-processing
(x2)

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

ProbeError

Readiness probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused body:

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Unhealthy

Readiness probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

ProbeError

Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Unhealthy

Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-ed80512f-5f4d667548-5d2rb

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-ed80512f-b5fbc8894-p9d48

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Killing

Stopping container istio-proxy

openshift-ingress

deployment-controller

e2e-derive-01750ee0-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-derive-01750ee0-openshift-default-58fb5bbddd from 0 to 1

openshift-ingress

replicaset-controller

e2e-derive-01750ee0-openshift-default-58fb5bbddd

SuccessfulCreate

Created pod: e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

openshift-ingress

gateway_labeler_controller

e2e-derive-01750ee0

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-derive-01750ee0

openshift-ingress

multus

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

AddedInterface

Add eth0 [10.133.0.53/23] from ovn-kubernetes

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-ait-ed80512f-openshift-default-7cbcf95486-h8s89

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

deployment-controller

payload-pre-processing-e2e-derive-01750ee0

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-derive-01750ee0-9765dc896 from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-derive-01750ee0-796ffc66fd

FailedCreate

Error creating: pods "payload-processing-e2e-derive-01750ee0-796ffc66fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-01750ee0: serviceaccount "payload-processing-e2e-derive-01750ee0" not found
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-derive-01750ee0-9765dc896

FailedCreate

Error creating: pods "payload-pre-processing-e2e-derive-01750ee0-9765dc896-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-01750ee0: serviceaccount "payload-processing-e2e-derive-01750ee0" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-derive-01750ee0-9765dc896

SuccessfulCreate

Created pod: payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

openshift-ingress

deployment-controller

payload-processing-e2e-derive-01750ee0

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-derive-01750ee0-796ffc66fd from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-derive-01750ee0-796ffc66fd

SuccessfulCreate

Created pod: payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

Started

Started container payload-pre-processing

openshift-ingress

multus

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

AddedInterface

Add eth0 [10.134.0.37/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

AddedInterface

Add eth0 [10.134.0.38/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

Created

Created container: payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x4)

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

ProbeError

Startup probe error: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Unhealthy

Startup probe failed: Get "http://10.133.0.53:15021/healthz/ready": dial tcp 10.133.0.53:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-01750ee0-9765dc896-8r98p

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-derive-01750ee0-796ffc66fd-plfjg

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-derive-01750ee0-openshift-default-58fb5bbddd-ms7f7

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

GatewayAuthBootstrap

e2e-mt-60d3c925

ReconcileError

Failed to reconcile EnvoyFilter: failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-mt-60d3c925-authn-ssl: envoyfilters.networking.istio.io "e2e-mt-60d3c925-authn-ssl" already exists

openshift-ingress

deployment-controller

e2e-mt-60d3c925-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-mt-60d3c925-openshift-default-595f945b68 from 0 to 1

openshift-ingress

replicaset-controller

e2e-mt-60d3c925-openshift-default-595f945b68

SuccessfulCreate

Created pod: e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

openshift-ingress

gateway_labeler_controller

e2e-mt-60d3c925

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-mt-60d3c925

openshift-ingress

multus

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

AddedInterface

Add eth0 [10.133.0.56/23] from ovn-kubernetes
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56

FailedCreate

Error creating: pods "payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-60d3c925: serviceaccount "payload-processing-e2e-mt-60d3c925" not found
(x6)

openshift-ingress

replicaset-controller

payload-processing-e2e-mt-60d3c925-96db7746

FailedCreate

Error creating: pods "payload-processing-e2e-mt-60d3c925-96db7746-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-60d3c925: serviceaccount "payload-processing-e2e-mt-60d3c925" not found

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56

SuccessfulCreate

Created pod: payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

openshift-ingress

deployment-controller

payload-processing-e2e-mt-60d3c925

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-mt-60d3c925-96db7746 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-mt-60d3c925

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56 from 0 to 1

openshift-ingress

kubelet

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

Started

Started container payload-processing

openshift-ingress

replicaset-controller

payload-processing-e2e-mt-60d3c925-96db7746

SuccessfulCreate

Created pod: payload-processing-e2e-mt-60d3c925-96db7746-6trnp

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

Created

Created container: payload-pre-processing

openshift-ingress

multus

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

AddedInterface

Add eth0 [10.134.0.40/23] from ovn-kubernetes

openshift-ingress

multus

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

AddedInterface

Add eth0 [10.134.0.41/23] from ovn-kubernetes
(x2)

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

ProbeError

Readiness probe error: Get "http://10.133.0.56:15021/healthz/ready": read tcp 10.133.0.2:41294->10.133.0.56:15021: read: connection reset by peer body:
(x2)

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Unhealthy

Readiness probe failed: Get "http://10.133.0.56:15021/healthz/ready": read tcp 10.133.0.2:41294->10.133.0.56:15021: read: connection reset by peer
(x2)

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Created

Created container: istio-proxy
(x5)

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Unhealthy

Startup probe failed: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

ProbeError

Startup probe error: Get "http://10.133.0.56:15021/healthz/ready": dial tcp 10.133.0.56:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-mt-60d3c925-96db7746-6trnp

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-60d3c925-6b9c65bf56-ttfrs

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-mt-60d3c925-openshift-default-595f945b68-hjp2t

Killing

Stopping container istio-proxy

openshift-ingress

deployment-controller

e2e-mt-b20ef80a-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-mt-b20ef80a-openshift-default-ddddd4c6d from 0 to 1

openshift-ingress

multus

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

AddedInterface

Add eth0 [10.133.0.58/23] from ovn-kubernetes

openshift-ingress

gateway_labeler_controller

e2e-mt-b20ef80a

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-mt-b20ef80a

openshift-ingress

replicaset-controller

e2e-mt-b20ef80a-openshift-default-ddddd4c6d

SuccessfulCreate

Created pod: e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

openshift-ingress

deployment-controller

payload-pre-processing-e2e-mt-b20ef80a

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d

FailedCreate

Error creating: pods "payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-b20ef80a: serviceaccount "payload-processing-e2e-mt-b20ef80a" not found

openshift-ingress

deployment-controller

payload-processing-e2e-mt-b20ef80a

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-mt-b20ef80a-dd556464b from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-mt-b20ef80a-dd556464b

SuccessfulCreate

Created pod: payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-mt-b20ef80a-dd556464b

FailedCreate

Error creating: pods "payload-processing-e2e-mt-b20ef80a-dd556464b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-mt-b20ef80a: serviceaccount "payload-processing-e2e-mt-b20ef80a" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

AddedInterface

Add eth0 [10.134.0.44/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d

SuccessfulCreate

Created pod: payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

openshift-ingress

kubelet

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

Started

Started container payload-processing

openshift-ingress

multus

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

AddedInterface

Add eth0 [10.134.0.43/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

Started

Started container payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Created

Created container: istio-proxy
(x4)

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

ProbeError

Startup probe error: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Unhealthy

Startup probe failed: Get "http://10.133.0.58:15021/healthz/ready": dial tcp 10.133.0.58:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-mt-b20ef80a-77fcd6c84d-wlcds

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-mt-b20ef80a-dd556464b-mwq2c

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

e2e-mt-b20ef80a-openshift-default-ddddd4c6d-wqwb4

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

gateway_labeler_controller

e2e-inf-a-4d9536

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-inf-a-4d9536

openshift-ingress

multus

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

AddedInterface

Add eth0 [10.133.0.61/23] from ovn-kubernetes

openshift-ingress

GatewayAuthBootstrap

e2e-inf-a-4d9536

ReconcileError

Failed to reconcile EnvoyFilter: failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-inf-a-4d9536-authn-ssl: envoyfilters.networking.istio.io "e2e-inf-a-4d9536-authn-ssl" already exists

openshift-ingress

deployment-controller

e2e-inf-a-4d9536-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-inf-a-4d9536-openshift-default-679df6499b from 0 to 1

openshift-ingress

replicaset-controller

e2e-inf-a-4d9536-openshift-default-679df6499b

SuccessfulCreate

Created pod: e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-inf-a-4d9536-6cd98cc99b

FailedCreate

Error creating: pods "payload-processing-e2e-inf-a-4d9536-6cd98cc99b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-4d9536: serviceaccount "payload-processing-e2e-inf-a-4d9536" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-inf-a-4d9536

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4 from 0 to 1

openshift-ingress

deployment-controller

payload-processing-e2e-inf-a-4d9536

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-inf-a-4d9536-6cd98cc99b from 0 to 1

openshift-ingress

kubelet

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

Started

Started container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

AddedInterface

Add eth0 [10.134.0.47/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4

SuccessfulCreate

Created pod: payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4

FailedCreate

Error creating: pods "payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-a-4d9536: serviceaccount "payload-processing-e2e-inf-a-4d9536" not found

openshift-ingress

multus

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

AddedInterface

Add eth0 [10.134.0.46/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-inf-a-4d9536-6cd98cc99b

SuccessfulCreate

Created pod: payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine
(x2)

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Unhealthy

Readiness probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

ProbeError

Readiness probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused body:
(x2)

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Created

Created container: istio-proxy
(x4)

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

ProbeError

Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Unhealthy

Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused

openshift-ingress

gateway_labeler_controller

e2e-inf-b-5c1453

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-inf-b-5c1453

openshift-ingress

deployment-controller

e2e-inf-b-5c1453-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-inf-b-5c1453-openshift-default-55d5855686 from 0 to 1

openshift-ingress

replicaset-controller

e2e-inf-b-5c1453-openshift-default-55d5855686

SuccessfulCreate

Created pod: e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

openshift-ingress

multus

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

AddedInterface

Add eth0 [10.133.0.63/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-processing-e2e-inf-b-5c1453

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-inf-b-5c1453-f467c8458 from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-inf-b-5c1453-f467c8458

FailedCreate

Error creating: pods "payload-processing-e2e-inf-b-5c1453-f467c8458-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-5c1453: serviceaccount "payload-processing-e2e-inf-b-5c1453" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-inf-b-5c1453

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76

SuccessfulCreate

Created pod: payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

openshift-ingress

replicaset-controller

payload-processing-e2e-inf-b-5c1453-f467c8458

SuccessfulCreate

Created pod: payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

openshift-ingress

kubelet

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

AddedInterface

Add eth0 [10.134.0.48/23] from ovn-kubernetes

openshift-ingress

multus

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

AddedInterface

Add eth0 [10.134.0.49/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

Started

Started container payload-pre-processing
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76

FailedCreate

Error creating: pods "payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-inf-b-5c1453: serviceaccount "payload-processing-e2e-inf-b-5c1453" not found

openshift-ingress

kubelet

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

Started

Started container payload-processing

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

ProbeError

Readiness probe error: Get "http://10.133.0.63:15021/healthz/ready": read tcp 10.133.0.2:58646->10.133.0.63:15021: read: connection reset by peer body:

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Unhealthy

Readiness probe failed: Get "http://10.133.0.63:15021/healthz/ready": read tcp 10.133.0.2:58646->10.133.0.63:15021: read: connection reset by peer
(x2)

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Started

Started container istio-proxy
(x4)

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Unhealthy

Startup probe failed: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

ProbeError

Startup probe error: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-inf-a-4d9536-6cd98cc99b-l7h9s

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-a-4d9536-7ffb776bd4-p26dn

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-inf-a-4d9536-openshift-default-679df6499b-kwfvn

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-pre-processing-e2e-inf-b-5c1453-76d96bbb76-7pc2k

Killing

Stopping container payload-pre-processing

openshift-ingress

endpoint-controller

payload-pre-processing-e2e-inf-b-5c1453

FailedToUpdateEndpoint

Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-inf-b-5c1453: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-inf-b-5c1453": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-inf-b-5c1453, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 979c0088-b749-4b69-9b85-d832a01c3a37, UID in object meta:

openshift-ingress

kubelet

payload-processing-e2e-inf-b-5c1453-f467c8458-z9nrb

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-inf-b-5c1453-openshift-default-55d5855686-p4bh8

Killing

Stopping container istio-proxy

openshift-ingress

gateway_labeler_controller

e2e-api-a-a62265

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-api-a-a62265

openshift-ingress

replicaset-controller

e2e-api-a-a62265-openshift-default-746d8ccd96

SuccessfulCreate

Created pod: e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

openshift-ingress

deployment-controller

e2e-api-a-a62265-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-api-a-a62265-openshift-default-746d8ccd96 from 0 to 1

openshift-ingress

multus

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

AddedInterface

Add eth0 [10.133.0.66/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-api-a-a62265-8cf98b75b

SuccessfulCreate

Created pod: payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-api-a-a62265-687f4f58d4

FailedCreate

Error creating: pods "payload-pre-processing-e2e-api-a-a62265-687f4f58d4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a62265: serviceaccount "payload-processing-e2e-api-a-a62265" not found

openshift-ingress

deployment-controller

payload-processing-e2e-api-a-a62265

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-api-a-a62265-8cf98b75b from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-api-a-a62265-8cf98b75b

FailedCreate

Error creating: pods "payload-processing-e2e-api-a-a62265-8cf98b75b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-a-a62265: serviceaccount "payload-processing-e2e-api-a-a62265" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-api-a-a62265

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-api-a-a62265-687f4f58d4 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-api-a-a62265-687f4f58d4

SuccessfulCreate

Created pod: payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

openshift-ingress

multus

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

AddedInterface

Add eth0 [10.134.0.53/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

AddedInterface

Add eth0 [10.134.0.54/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

Started

Started container payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Created

Created container: istio-proxy
(x4)

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Unhealthy

Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

ProbeError

Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused body:

openshift-ingress

deployment-controller

e2e-api-b-3adaab-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-api-b-3adaab-openshift-default-659664b78 from 0 to 1

openshift-ingress

replicaset-controller

e2e-api-b-3adaab-openshift-default-659664b78

SuccessfulCreate

Created pod: e2e-api-b-3adaab-openshift-default-659664b78-l8skx

openshift-ingress

gateway_labeler_controller

e2e-api-b-3adaab

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-api-b-3adaab

openshift-ingress

multus

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

AddedInterface

Add eth0 [10.133.0.68/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-processing-e2e-api-b-3adaab

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-api-b-3adaab-8566476754 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-api-b-3adaab

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-api-b-3adaab-678dfc95bd from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-api-b-3adaab-8566476754

FailedCreate

Error creating: pods "payload-processing-e2e-api-b-3adaab-8566476754-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-3adaab: serviceaccount "payload-processing-e2e-api-b-3adaab" not found

openshift-ingress

kubelet

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

Started

Started container payload-processing

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd

SuccessfulCreate

Created pod: payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

openshift-ingress

kubelet

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

Created

Created container: payload-pre-processing
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd

FailedCreate

Error creating: pods "payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-api-b-3adaab: serviceaccount "payload-processing-e2e-api-b-3adaab" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

AddedInterface

Add eth0 [10.134.0.55/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

AddedInterface

Add eth0 [10.134.0.56/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-api-b-3adaab-8566476754

SuccessfulCreate

Created pod: payload-processing-e2e-api-b-3adaab-8566476754-tv72x

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

ProbeError

Readiness probe error: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused body:
(x2)

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Unhealthy

Readiness probe failed: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

ProbeError

Startup probe error: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Unhealthy

Startup probe failed: Get "http://10.133.0.68:15021/healthz/ready": dial tcp 10.133.0.68:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-api-a-a62265-687f4f58d4-4rt8w

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-api-a-a62265-8cf98b75b-gx5tx

Killing

Stopping container payload-processing

openshift-ingress

endpoint-controller

payload-pre-processing-e2e-api-a-a62265

FailedToUpdateEndpoint

Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-api-a-a62265: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-api-a-a62265": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-api-a-a62265, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8cfce0bf-6d6d-46a8-8b71-7e7874744542, UID in object meta:

openshift-ingress

kubelet

e2e-api-a-a62265-openshift-default-746d8ccd96-8d72c

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-pre-processing-e2e-api-b-3adaab-678dfc95bd-kj6f5

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-api-b-3adaab-8566476754-tv72x

Killing

Stopping container payload-processing

openshift-ingress

gateway_labeler_controller

e2e-shared-a-9a3702

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-shared-a-9a3702

openshift-ingress

deployment-controller

e2e-shared-a-9a3702-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-shared-a-9a3702-openshift-default-6d9c9f9f6 from 0 to 1

openshift-ingress

replicaset-controller

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6

SuccessfulCreate

Created pod: e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

openshift-ingress

kubelet

e2e-api-b-3adaab-openshift-default-659664b78-l8skx

Killing

Stopping container istio-proxy

openshift-ingress

multus

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

AddedInterface

Add eth0 [10.133.0.70/23] from ovn-kubernetes
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-shared-a-9a3702-5455bf49d

FailedCreate

Error creating: pods "payload-processing-e2e-shared-a-9a3702-5455bf49d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-9a3702: serviceaccount "payload-processing-e2e-shared-a-9a3702" not found

openshift-ingress

deployment-controller

payload-processing-e2e-shared-a-9a3702

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-shared-a-9a3702-5455bf49d from 0 to 1

openshift-ingress

multus

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

AddedInterface

Add eth0 [10.134.0.59/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-pre-processing-e2e-shared-a-9a3702

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-shared-a-9a3702-844489479b from 0 to 1

openshift-ingress

multus

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

AddedInterface

Add eth0 [10.134.0.60/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

replicaset-controller

payload-processing-e2e-shared-a-9a3702-5455bf49d

SuccessfulCreate

Created pod: payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-shared-a-9a3702-844489479b

SuccessfulCreate

Created pod: payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-shared-a-9a3702-844489479b

FailedCreate

Error creating: pods "payload-pre-processing-e2e-shared-a-9a3702-844489479b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-a-9a3702: serviceaccount "payload-processing-e2e-shared-a-9a3702" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

Created

Created container: payload-processing
(x2)

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Created

Created container: istio-proxy

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Unhealthy

Readiness probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

ProbeError

Readiness probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

ProbeError

Startup probe error: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Unhealthy

Startup probe failed: Get "http://10.133.0.70:15021/healthz/ready": dial tcp 10.133.0.70:15021: connect: connection refused

openshift-ingress

replicaset-controller

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd

SuccessfulCreate

Created pod: e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

openshift-ingress

multus

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

AddedInterface

Add eth0 [10.133.0.72/23] from ovn-kubernetes

openshift-ingress

gateway_labeler_controller

e2e-shared-b-8ef216

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-shared-b-8ef216

openshift-ingress

deployment-controller

e2e-shared-b-8ef216-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-shared-b-8ef216-openshift-default-5dfdfd99dd from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-shared-b-8ef216-545cb78556

SuccessfulCreate

Created pod: payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

openshift-ingress

deployment-controller

payload-pre-processing-e2e-shared-b-8ef216

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-shared-b-8ef216-545cb78556 from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-shared-b-8ef216-545cb78556

FailedCreate

Error creating: pods "payload-pre-processing-e2e-shared-b-8ef216-545cb78556-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-8ef216: serviceaccount "payload-processing-e2e-shared-b-8ef216" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b

SuccessfulCreate

Created pod: payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

openshift-ingress

deployment-controller

payload-processing-e2e-shared-b-8ef216

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-shared-b-8ef216-54dd6d8b8b from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b

FailedCreate

Error creating: pods "payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-shared-b-8ef216: serviceaccount "payload-processing-e2e-shared-b-8ef216" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

AddedInterface

Add eth0 [10.134.0.62/23] from ovn-kubernetes

openshift-ingress

multus

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

AddedInterface

Add eth0 [10.134.0.61/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

Started

Started container payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Unhealthy

Readiness probe failed: Get "http://10.133.0.72:15021/healthz/ready": read tcp 10.133.0.2:35878->10.133.0.72:15021: read: connection reset by peer

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

ProbeError

Readiness probe error: Get "http://10.133.0.72:15021/healthz/ready": read tcp 10.133.0.2:35878->10.133.0.72:15021: read: connection reset by peer body:
(x2)

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Created

Created container: istio-proxy
(x5)

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

ProbeError

Startup probe error: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Unhealthy

Startup probe failed: Get "http://10.133.0.72:15021/healthz/ready": dial tcp 10.133.0.72:15021: connect: connection refused

openshift-ingress

gateway_labeler_controller

e2e-ipp-a-d722d0

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-a-d722d0

openshift-ingress

replicaset-controller

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d

SuccessfulCreate

Created pod: e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

openshift-ingress

deployment-controller

e2e-ipp-a-d722d0-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ipp-a-d722d0-openshift-default-5bb568fc4d from 0 to 1

openshift-ingress

multus

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

AddedInterface

Add eth0 [10.133.0.79/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-a-d722d0-d7879bc5b

SuccessfulCreate

Created pod: payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-d722d0: serviceaccount "payload-processing-e2e-ipp-a-d722d0" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ipp-a-d722d0

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6 from 0 to 1

openshift-ingress

deployment-controller

payload-processing-e2e-ipp-a-d722d0

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ipp-a-d722d0-d7879bc5b from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr
(x6)

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-a-d722d0-d7879bc5b

FailedCreate

Error creating: pods "payload-processing-e2e-ipp-a-d722d0-d7879bc5b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-a-d722d0: serviceaccount "payload-processing-e2e-ipp-a-d722d0" not found

openshift-ingress

kubelet

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

Started

Started container payload-processing

openshift-ingress

multus

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

AddedInterface

Add eth0 [10.134.0.67/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

AddedInterface

Add eth0 [10.134.0.68/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

Started

Started container payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x3)

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

ProbeError

Startup probe error: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused body:
(x3)

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Unhealthy

Startup probe failed: Get "http://10.133.0.79:15021/healthz/ready": dial tcp 10.133.0.79:15021: connect: connection refused

openshift-ingress

gateway_labeler_controller

e2e-ipp-b-5560c5

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-b-5560c5

openshift-ingress

deployment-controller

e2e-ipp-b-5560c5-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96 from 0 to 1

openshift-ingress

multus

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

AddedInterface

Add eth0 [10.132.0.57/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96

SuccessfulCreate

Created pod: e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5560c5: serviceaccount "payload-processing-e2e-ipp-b-5560c5" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ipp-b-5560c5

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588 from 0 to 1

openshift-ingress

deployment-controller

payload-processing-e2e-ipp-b-5560c5

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ipp-b-5560c5-75b9958d8d from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-b-5560c5-75b9958d8d

SuccessfulCreate

Created pod: payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-b-5560c5-75b9958d8d

FailedCreate

Error creating: pods "payload-processing-e2e-ipp-b-5560c5-75b9958d8d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-b-5560c5: serviceaccount "payload-processing-e2e-ipp-b-5560c5" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

Created

Created container: payload-pre-processing

openshift-ingress

multus

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

AddedInterface

Add eth0 [10.134.0.69/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

AddedInterface

Add eth0 [10.134.0.70/23] from ovn-kubernetes
(x2)

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x5)

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

ProbeError

Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Unhealthy

Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused

openshift-ingress

deployment-controller

e2e-ipp-cleanup-ebceb0-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b from 0 to 1

openshift-ingress

GatewayAuthBootstrap

e2e-ipp-cleanup-ebceb0

ReconcileError

Failed to reconcile EnvoyFilter: failed to create EnvoyFilter: could not CREATE EnvoyFilter openshift-ingress/e2e-ipp-cleanup-ebceb0-authn-ssl: envoyfilters.networking.istio.io "e2e-ipp-cleanup-ebceb0-authn-ssl" already exists

openshift-ingress

gateway_labeler_controller

e2e-ipp-cleanup-ebceb0

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ipp-cleanup-ebceb0

openshift-ingress

replicaset-controller

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b

SuccessfulCreate

Created pod: e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

openshift-ingress

multus

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

AddedInterface

Add eth0 [10.133.0.80/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86

SuccessfulCreate

Created pod: payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86

FailedCreate

Error creating: pods "payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-ebceb0: serviceaccount "payload-processing-e2e-ipp-cleanup-ebceb0" not found

openshift-ingress

deployment-controller

payload-processing-e2e-ipp-cleanup-ebceb0

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86 from 0 to 1

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ipp-cleanup-ebceb0

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ipp-cleanup-ebceb0: serviceaccount "payload-processing-e2e-ipp-cleanup-ebceb0" not found

openshift-ingress

kubelet

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

Started

Started container payload-pre-processing

openshift-ingress

multus

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

AddedInterface

Add eth0 [10.134.0.72/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

Created

Created container: payload-processing

openshift-ingress

multus

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

AddedInterface

Add eth0 [10.134.0.73/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

Started

Started container payload-processing
(x2)

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x3)

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Unhealthy

Startup probe failed: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused
(x3)

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

ProbeError

Startup probe error: Get "http://10.133.0.80:15021/healthz/ready": dial tcp 10.133.0.80:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-ipp-cleanup-ebceb0-769d468c86-4fzgv

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-cleanup-ebceb0-866dddddf6-rgspw

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

e2e-ipp-cleanup-ebceb0-openshift-default-d984d776b-qdxkq

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

payload-processing-e2e-ipp-a-d722d0-d7879bc5b-kvqws

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-a-d722d0-6f7849bf6-jd4wr

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-ipp-a-d722d0-openshift-default-5bb568fc4d-8ffvt

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-processing-e2e-ipp-b-5560c5-75b9958d8d-qz674

Killing

Stopping container payload-processing

openshift-ingress

endpoint-controller

payload-pre-processing-e2e-ipp-b-5560c5

FailedToUpdateEndpoint

Failed to update endpoint openshift-ingress/payload-pre-processing-e2e-ipp-b-5560c5: Operation cannot be fulfilled on endpoints "payload-pre-processing-e2e-ipp-b-5560c5": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/openshift-ingress/payload-pre-processing-e2e-ipp-b-5560c5, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: de43cf41-45fa-4c2d-beac-60a2c441bb3c, UID in object meta:

openshift-ingress

kubelet

payload-pre-processing-e2e-ipp-b-5560c5-6b6c586588-ndv4r

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-ipp-b-5560c5-openshift-default-7c5dfd6d96-wmqdr

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-processing-e2e-shared-a-9a3702-5455bf49d-gwbr4

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-a-9a3702-844489479b-4n6v2

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-shared-a-9a3702-openshift-default-6d9c9f9f6-mb472

Killing

Stopping container istio-proxy

openshift-ingress

kubelet

payload-processing-e2e-shared-b-8ef216-54dd6d8b8b-6zlmd

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-shared-b-8ef216-545cb78556-gmd65

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-shared-b-8ef216-openshift-default-5dfdfd99dd-5l7gv

Killing

Stopping container istio-proxy