Time Namespace Component RelatedObject Reason Message

openshift-ingress

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq to ip-10-0-137-105.ec2.internal

openshift-ingress

maas-default-gateway-openshift-default-5ccb968c66-njznb

Scheduled

Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-njznb to ip-10-0-141-115.ec2.internal

openshift-ingress

kube-auth-proxy-6bd989cf94-wrrd6

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-6bd989cf94-wrrd6 to ip-10-0-137-105.ec2.internal

openshift-ingress

kube-auth-proxy-6bd989cf94-psc42

Scheduled

Successfully assigned openshift-ingress/kube-auth-proxy-6bd989cf94-psc42 to ip-10-0-141-115.ec2.internal

openshift-ingress

istiod-openshift-gateway-7b7995d5cd-zpmz8

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zpmz8 to ip-10-0-141-115.ec2.internal

openshift-ingress

istiod-openshift-gateway-5677ccb49b-qh57b

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-qh57b to ip-10-0-136-206.ec2.internal

openshift-ingress

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Scheduled

Successfully assigned openshift-ingress/e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6 to ip-10-0-141-115.ec2.internal

openshift-ingress

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54 to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-pre-processing-687dbb6cf6-7mkdk

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-687dbb6cf6-7mkdk to ip-10-0-137-105.ec2.internal

openshift-ingress

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

Scheduled

Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd to ip-10-0-141-115.ec2.internal

openshift-ingress

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Scheduled

Successfully assigned openshift-ingress/e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt to ip-10-0-141-115.ec2.internal

openshift-ingress

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Scheduled

Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6 to ip-10-0-141-115.ec2.internal

openshift-ingress

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

Scheduled

Successfully assigned openshift-ingress/payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5 to ip-10-0-137-105.ec2.internal

openshift-ingress

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Scheduled

Successfully assigned openshift-ingress/e2e-derive-842768d6-openshift-default-554998dc56-5plcs to ip-10-0-141-115.ec2.internal

openshift-ingress

payload-processing-5cc6cd444b-gq4hn

Scheduled

Successfully assigned openshift-ingress/payload-processing-5cc6cd444b-gq4hn to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc to ip-10-0-137-105.ec2.internal

openshift-ingress

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

Scheduled

Successfully assigned openshift-ingress/payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487 to ip-10-0-137-105.ec2.internal

openshift-ingress

router-default-78549c97ff-82xwq

Scheduled

Successfully assigned openshift-ingress/router-default-78549c97ff-82xwq to ip-10-0-136-206.ec2.internal

openshift-ingress

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Scheduled

Successfully assigned openshift-ingress/e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh to ip-10-0-141-115.ec2.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-78549c97ff from 0 to 1

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x16)

openshift-ingress

replicaset-controller

router-default-78549c97ff

FailedCreate

Error creating: pods "router-default-78549c97ff-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts

openshift-ingress

replicaset-controller

router-default-78549c97ff

SuccessfulCreate

Created pod: router-default-78549c97ff-82xwq
(x6)

openshift-ingress

kubelet

router-default-78549c97ff-82xwq

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt
(x6)

openshift-ingress

kubelet

router-default-78549c97ff-82xwq

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found

openshift-ingress

kubelet

router-default-78549c97ff-82xwq

Started

Started container router

openshift-ingress

kubelet

router-default-78549c97ff-82xwq

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559" already present on machine

openshift-ingress

multus

router-default-78549c97ff-82xwq

AddedInterface

Add eth0 [10.132.0.11/23] from ovn-kubernetes

openshift-ingress

kubelet

router-default-78549c97ff-82xwq

Created

Created container: router

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-wrrd6

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-psc42

Pulling

Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2"
(x4)

openshift-ingress

replicaset-controller

kube-auth-proxy-6bd989cf94

FailedCreate

Error creating: pods "kube-auth-proxy-6bd989cf94-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found

openshift-ingress

replicaset-controller

kube-auth-proxy-6bd989cf94

SuccessfulCreate

Created pod: kube-auth-proxy-6bd989cf94-psc42

openshift-ingress

replicaset-controller

kube-auth-proxy-6bd989cf94

SuccessfulCreate

Created pod: kube-auth-proxy-6bd989cf94-wrrd6

openshift-ingress

deployment-controller

kube-auth-proxy

ScalingReplicaSet

Scaled up replica set kube-auth-proxy-6bd989cf94 from 0 to 2

openshift-ingress

multus

kube-auth-proxy-6bd989cf94-wrrd6

AddedInterface

Add eth0 [10.133.0.19/23] from ovn-kubernetes

openshift-ingress

multus

kube-auth-proxy-6bd989cf94-psc42

AddedInterface

Add eth0 [10.134.0.27/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7b7995d5cd

SuccessfulCreate

Created pod: istiod-openshift-gateway-7b7995d5cd-zpmz8

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"

openshift-ingress

multus

istiod-openshift-gateway-7b7995d5cd-zpmz8

AddedInterface

Add eth0 [10.134.0.30/23] from ovn-kubernetes

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-wrrd6

Created

Created container: kube-auth-proxy

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-wrrd6

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.564s (2.564s including waiting). Image size: 265120132 bytes.

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-wrrd6

Started

Started container kube-auth-proxy

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-psc42

Started

Started container kube-auth-proxy

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-psc42

Pulled

Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 4.655s (4.655s including waiting). Image size: 265120132 bytes.

openshift-ingress

kubelet

kube-auth-proxy-6bd989cf94-psc42

Created

Created container: kube-auth-proxy

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 4.74s (4.74s including waiting). Image size: 181126104 bytes.

openshift-ingress

deployment-controller

data-science-gateway-data-science-gateway-class

ScalingReplicaSet

Scaled up replica set data-science-gateway-data-science-gateway-class-fddd6c88c from 0 to 1

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-qh57b

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31"

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"

openshift-ingress

replicaset-controller

data-science-gateway-data-science-gateway-class-68cf4d799c

SuccessfulCreate

Created pod: data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

openshift-ingress

multus

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

AddedInterface

Add eth0 [10.134.0.31/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

data-science-gateway-data-science-gateway-class-fddd6c88c

SuccessfulCreate

Created pod: data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

openshift-ingress

replicaset-controller

istiod-openshift-gateway-5677ccb49b

SuccessfulCreate

Created pod: istiod-openshift-gateway-5677ccb49b-qh57b

openshift-ingress

deployment-controller

data-science-gateway-data-science-gateway-class

ScalingReplicaSet

Scaled up replica set data-science-gateway-data-science-gateway-class-68cf4d799c from 0 to 1

openshift-ingress

multus

istiod-openshift-gateway-5677ccb49b-qh57b

AddedInterface

Add eth0 [10.132.0.30/23] from ovn-kubernetes

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec"

openshift-ingress

multus

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

AddedInterface

Add eth0 [10.134.0.32/23] from ovn-kubernetes

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

Created

Created container: istio-proxy

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.936s (2.936s including waiting). Image size: 181318953 bytes.

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Started

Started container istio-proxy

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-qh57b

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.592s (2.592s including waiting). Image size: 181126104 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-qh57b

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-5677ccb49b-qh57b

Started

Started container discovery

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

Started

Started container istio-proxy

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-68cf4d799ch7nzd

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 2.651s (2.651s including waiting). Image size: 181318953 bytes.

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Created

Created container: istio-proxy

openshift-ingress

kubelet

istiod-openshift-gateway-7b7995d5cd-zpmz8

Killing

Stopping container discovery

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7b7995d5cd

SuccessfulDelete

Deleted pod: istiod-openshift-gateway-7b7995d5cd-zpmz8

openshift-ingress

replicaset-controller

data-science-gateway-data-science-gateway-class-fddd6c88c

SuccessfulDelete

Deleted pod: data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

openshift-ingress

deployment-controller

data-science-gateway-data-science-gateway-class

ScalingReplicaSet

Scaled down replica set data-science-gateway-data-science-gateway-class-fddd6c88c from 1 to 0
(x3)

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Unhealthy

Startup probe failed: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused
(x3)

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

ProbeError

Startup probe error: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused body:

openshift-ingress

kubelet

data-science-gateway-data-science-gateway-class-fddd6c88c-8gbv6

Killing

Stopping container istio-proxy

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

openshift-ingress

horizontal-pod-autoscaler

kube-auth-proxy

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

openshift-ingress

replicaset-controller

maas-default-gateway-openshift-default-5ccb968c66

SuccessfulCreate

Created pod: maas-default-gateway-openshift-default-5ccb968c66-njznb

openshift-ingress

deployment-controller

maas-default-gateway-openshift-default

ScalingReplicaSet

Scaled up replica set maas-default-gateway-openshift-default-5ccb968c66 from 0 to 1

openshift-ingress

gateway_labeler_controller

maas-default-gateway

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway

openshift-ingress

multus

maas-default-gateway-openshift-default-5ccb968c66-njznb

AddedInterface

Add eth0 [10.134.0.50/23] from ovn-kubernetes
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-687dbb6cf6

FailedCreate

Error creating: pods "payload-pre-processing-687dbb6cf6-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found

openshift-ingress

deployment-controller

payload-pre-processing

ScalingReplicaSet

Scaled up replica set payload-pre-processing-687dbb6cf6 from 0 to 1
(x6)

openshift-ingress

replicaset-controller

payload-processing-5cc6cd444b

FailedCreate

Error creating: pods "payload-processing-5cc6cd444b-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found

openshift-ingress

deployment-controller

payload-processing

ScalingReplicaSet

Scaled up replica set payload-processing-5cc6cd444b from 0 to 1

openshift-ingress

multus

payload-processing-5cc6cd444b-gq4hn

AddedInterface

Add eth0 [10.133.0.29/23] from ovn-kubernetes

openshift-ingress

multus

payload-pre-processing-687dbb6cf6-7mkdk

AddedInterface

Add eth0 [10.133.0.28/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-5cc6cd444b-gq4hn

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"

openshift-ingress

replicaset-controller

payload-processing-5cc6cd444b

SuccessfulCreate

Created pod: payload-processing-5cc6cd444b-gq4hn

openshift-ingress

replicaset-controller

payload-pre-processing-687dbb6cf6

SuccessfulCreate

Created pod: payload-pre-processing-687dbb6cf6-7mkdk

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-7mkdk

Pulling

Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable"

openshift-ingress

kubelet

payload-processing-5cc6cd444b-gq4hn

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.551s (2.551s including waiting). Image size: 165776054 bytes.

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-7mkdk

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-7mkdk

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-687dbb6cf6-7mkdk

Pulled

Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.724s (2.724s including waiting). Image size: 165776054 bytes.

openshift-ingress

kubelet

payload-processing-5cc6cd444b-gq4hn

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-5cc6cd444b-gq4hn

Started

Started container payload-processing
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-5ccb968c66-njznb

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-5ccb968c66-njznb

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-5ccb968c66-njznb

Started

Started container istio-proxy
(x4)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-5ccb968c66-njznb

ProbeError

Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

maas-default-gateway-openshift-default-5ccb968c66-njznb

Unhealthy

Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused

openshift-ingress

multus

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

AddedInterface

Add eth0 [10.134.0.57/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

e2e-ait-3102a353-openshift-default-7cf5d684c6

SuccessfulCreate

Created pod: e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

openshift-ingress

deployment-controller

e2e-ait-3102a353-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ait-3102a353-openshift-default-7cf5d684c6 from 0 to 1

openshift-ingress

gateway_labeler_controller

e2e-ait-3102a353

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ait-3102a353

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-3102a353-59c5d469cb

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

openshift-ingress

deployment-controller

payload-processing-e2e-ait-3102a353

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ait-3102a353-fd576ddd8 from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-3102a353-fd576ddd8

FailedCreate

Error creating: pods "payload-processing-e2e-ait-3102a353-fd576ddd8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-3102a353: serviceaccount "payload-processing-e2e-ait-3102a353" not found

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ait-3102a353

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ait-3102a353-59c5d469cb from 0 to 1
(x4)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-3102a353-59c5d469cb

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ait-3102a353-59c5d469cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-3102a353: serviceaccount "payload-processing-e2e-ait-3102a353" not found

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-3102a353-fd576ddd8

SuccessfulCreate

Created pod: payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

openshift-ingress

kubelet

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

Created

Created container: payload-processing

openshift-ingress

multus

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

AddedInterface

Add eth0 [10.133.0.38/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

AddedInterface

Add eth0 [10.133.0.39/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

ProbeError

Readiness probe error: Get "http://10.134.0.57:15021/healthz/ready": read tcp 10.134.0.2:37602->10.134.0.57:15021: read: connection reset by peer body:

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Unhealthy

Readiness probe failed: Get "http://10.134.0.57:15021/healthz/ready": read tcp 10.134.0.2:37602->10.134.0.57:15021: read: connection reset by peer
(x2)

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Created

Created container: istio-proxy
(x3)

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Unhealthy

Startup probe failed: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused
(x3)

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

ProbeError

Startup probe error: Get "http://10.134.0.57:15021/healthz/ready": dial tcp 10.134.0.57:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-ait-3102a353-fd576ddd8-s24gc

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-3102a353-59c5d469cb-nhrsz

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-ait-3102a353-openshift-default-7cf5d684c6-gpdzt

Killing

Stopping container istio-proxy

openshift-ingress

gateway_labeler_controller

e2e-migrate-54f231e4-gw

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-migrate-54f231e4-gw

openshift-ingress

deployment-controller

e2e-migrate-54f231e4-gw-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6 from 0 to 1

openshift-ingress

replicaset-controller

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6

SuccessfulCreate

Created pod: e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

openshift-ingress

multus

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

AddedInterface

Add eth0 [10.134.0.58/23] from ovn-kubernetes
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-migrate-54f231e4-fd5bb66c7

FailedCreate

Error creating: pods "payload-processing-e2e-migrate-54f231e4-fd5bb66c7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-54f231e4: serviceaccount "payload-processing-e2e-migrate-54f231e4" not found

openshift-ingress

deployment-controller

payload-processing-e2e-migrate-54f231e4

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-migrate-54f231e4-fd5bb66c7 from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-migrate-54f231e4-fd5bb66c7

SuccessfulCreate

Created pod: payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

openshift-ingress

deployment-controller

payload-pre-processing-e2e-migrate-54f231e4

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-migrate-54f231e4-6d49c5888 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888

SuccessfulCreate

Created pod: payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888

FailedCreate

Error creating: pods "payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-migrate-54f231e4: serviceaccount "payload-processing-e2e-migrate-54f231e4" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

Started

Started container payload-pre-processing

openshift-ingress

multus

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

AddedInterface

Add eth0 [10.133.0.41/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

Started

Started container payload-processing

openshift-ingress

multus

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

AddedInterface

Add eth0 [10.133.0.42/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

Created

Created container: payload-pre-processing
(x2)

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Created

Created container: istio-proxy

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Unhealthy

Readiness probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

ProbeError

Readiness probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

ProbeError

Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused body:
(x5)

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Unhealthy

Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused

openshift-ingress

kubelet

payload-pre-processing-e2e-migrate-54f231e4-6d49c5888-js4g5

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-migrate-54f231e4-fd5bb66c7-gn487

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-migrate-54f231e4-gw-openshift-default-986bdbdc6-wdff6

Killing

Stopping container istio-proxy

openshift-ingress

gateway_labeler_controller

e2e-ait-20962f79

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-ait-20962f79

openshift-ingress

deployment-controller

e2e-ait-20962f79-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-ait-20962f79-openshift-default-6578f7c54 from 0 to 1

openshift-ingress

replicaset-controller

e2e-ait-20962f79-openshift-default-6578f7c54

SuccessfulCreate

Created pod: e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

openshift-ingress

multus

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

AddedInterface

Add eth0 [10.134.0.59/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-20962f79-7d75b7787b

SuccessfulCreate

Created pod: payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq
(x6)

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-20962f79-6894c6cdb

FailedCreate

Error creating: pods "payload-processing-e2e-ait-20962f79-6894c6cdb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-20962f79: serviceaccount "payload-processing-e2e-ait-20962f79" not found

openshift-ingress

deployment-controller

payload-processing-e2e-ait-20962f79

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-ait-20962f79-6894c6cdb from 0 to 1

openshift-ingress

replicaset-controller

payload-processing-e2e-ait-20962f79-6894c6cdb

SuccessfulCreate

Created pod: payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

openshift-ingress

deployment-controller

payload-pre-processing-e2e-ait-20962f79

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-ait-20962f79-7d75b7787b from 0 to 1
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-ait-20962f79-7d75b7787b

FailedCreate

Error creating: pods "payload-pre-processing-e2e-ait-20962f79-7d75b7787b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-ait-20962f79: serviceaccount "payload-processing-e2e-ait-20962f79" not found

openshift-ingress

kubelet

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

Started

Started container payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

AddedInterface

Add eth0 [10.133.0.44/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

Started

Started container payload-pre-processing

openshift-ingress

multus

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

AddedInterface

Add eth0 [10.133.0.45/23] from ovn-kubernetes
(x2)

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Started

Started container istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x5)

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Unhealthy

Startup probe failed: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused
(x5)

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

ProbeError

Startup probe error: Get "http://10.134.0.59:15021/healthz/ready": dial tcp 10.134.0.59:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-processing-e2e-ait-20962f79-6894c6cdb-2mwhm

Killing

Stopping container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-ait-20962f79-7d75b7787b-9kgjq

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

e2e-ait-20962f79-openshift-default-6578f7c54-rxlmh

Killing

Stopping container istio-proxy

openshift-ingress

gateway_labeler_controller

e2e-derive-842768d6

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway e2e-derive-842768d6

openshift-ingress

replicaset-controller

e2e-derive-842768d6-openshift-default-554998dc56

SuccessfulCreate

Created pod: e2e-derive-842768d6-openshift-default-554998dc56-5plcs

openshift-ingress

deployment-controller

e2e-derive-842768d6-openshift-default

ScalingReplicaSet

Scaled up replica set e2e-derive-842768d6-openshift-default-554998dc56 from 0 to 1

openshift-ingress

multus

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

AddedInterface

Add eth0 [10.134.0.60/23] from ovn-kubernetes

openshift-ingress

deployment-controller

payload-pre-processing-e2e-derive-842768d6

ScalingReplicaSet

Scaled up replica set payload-pre-processing-e2e-derive-842768d6-66cbdc65bd from 0 to 1

openshift-ingress

deployment-controller

payload-processing-e2e-derive-842768d6

ScalingReplicaSet

Scaled up replica set payload-processing-e2e-derive-842768d6-859c5565c8 from 0 to 1

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd

SuccessfulCreate

Created pod: payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

openshift-ingress

replicaset-controller

payload-processing-e2e-derive-842768d6-859c5565c8

SuccessfulCreate

Created pod: payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc
(x5)

openshift-ingress

replicaset-controller

payload-processing-e2e-derive-842768d6-859c5565c8

FailedCreate

Error creating: pods "payload-processing-e2e-derive-842768d6-859c5565c8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-842768d6: serviceaccount "payload-processing-e2e-derive-842768d6" not found
(x5)

openshift-ingress

replicaset-controller

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd

FailedCreate

Error creating: pods "payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-derive-842768d6: serviceaccount "payload-processing-e2e-derive-842768d6" not found

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

AddedInterface

Add eth0 [10.133.0.47/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

Pulled

Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine

openshift-ingress

multus

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

openshift-ingress

kubelet

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

Created

Created container: payload-processing

openshift-ingress

kubelet

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

Started

Started container payload-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Created

Created container: payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Started

Started container payload-pre-processing

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Unhealthy

Readiness probe failed: dial tcp 10.133.0.48:9004: connect: connection refused
(x2)

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine
(x2)

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Created

Created container: istio-proxy
(x2)

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Started

Started container istio-proxy

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Unhealthy

Readiness probe failed: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

ProbeError

Readiness probe error: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused body:
(x4)

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Unhealthy

Startup probe failed: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused
(x4)

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

ProbeError

Startup probe error: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused body:

openshift-ingress

kubelet

payload-pre-processing-e2e-derive-842768d6-66cbdc65bd-bcc54

Killing

Stopping container payload-pre-processing

openshift-ingress

kubelet

payload-processing-e2e-derive-842768d6-859c5565c8-lfxgc

Killing

Stopping container payload-processing

openshift-ingress

kubelet

e2e-derive-842768d6-openshift-default-554998dc56-5plcs

Killing

Stopping container istio-proxy