=== events in openshift-ingress (2026-09-19T04:29:33Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 7m36s Normal Scheduled pod/kube-auth-proxy-77df8579b6-57ph6 Successfully assigned openshift-ingress/kube-auth-proxy-77df8579b6-57ph6 to ip-10-0-138-12.ec2.internal 7m28s Normal Scheduled pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 to ip-10-0-136-23.ec2.internal 3m46s Normal Scheduled pod/payload-processing-5b54b7fbb7-pqcb8 Successfully assigned openshift-ingress/payload-processing-5b54b7fbb7-pqcb8 to ip-10-0-138-12.ec2.internal 13m Normal Scheduled pod/router-default-9c66fb948-4h58s Successfully assigned openshift-ingress/router-default-9c66fb948-4h58s to ip-10-0-132-168.ec2.internal 7m36s Normal Scheduled pod/kube-auth-proxy-77df8579b6-fd2ss Successfully assigned openshift-ingress/kube-auth-proxy-77df8579b6-fd2ss to ip-10-0-132-168.ec2.internal 5m4s Normal Scheduled pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-c747fcbf8-m7vfh to ip-10-0-136-23.ec2.internal 3m46s Normal Scheduled pod/payload-pre-processing-59fbc47bf8-9v8gw Successfully assigned openshift-ingress/payload-pre-processing-59fbc47bf8-9v8gw to ip-10-0-132-168.ec2.internal 35s Normal Scheduled pod/payload-processing-5b54b7fbb7-l487m Successfully assigned openshift-ingress/payload-processing-5b54b7fbb7-l487m to ip-10-0-138-12.ec2.internal 7m18s Normal Scheduled pod/istiod-openshift-gateway-5677ccb49b-dr5l5 Successfully assigned openshift-ingress/istiod-openshift-gateway-5677ccb49b-dr5l5 to ip-10-0-138-12.ec2.internal 35s Normal Scheduled pod/payload-pre-processing-59fbc47bf8-z5v5b Successfully assigned openshift-ingress/payload-pre-processing-59fbc47bf8-z5v5b to ip-10-0-138-12.ec2.internal 7m34s Normal Scheduled pod/istiod-openshift-gateway-7b7995d5cd-52gjw Successfully assigned openshift-ingress/istiod-openshift-gateway-7b7995d5cd-52gjw to ip-10-0-138-12.ec2.internal 19m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-9c66fb948 from 0 to 1 19m Warning FailedCreate replicaset/router-default-9c66fb948 Error creating: pods "router-default-9c66fb948-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 18m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 18m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 18m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 16m Warning FailedCreate replicaset/router-default-9c66fb948 Error creating: pods "router-default-9c66fb948-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes 13m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 13m Normal SuccessfulCreate replicaset/router-default-9c66fb948 Created pod: router-default-9c66fb948-4h58s 13m Warning FailedMount pod/router-default-9c66fb948-4h58s MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 13m Warning FailedMount pod/router-default-9c66fb948-4h58s MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 12m Normal Started pod/router-default-9c66fb948-4h58s Started container router 12m Normal Created pod/router-default-9c66fb948-4h58s Created container: router 12m Normal Pulled pod/router-default-9c66fb948-4h58s Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eaa467bc105be2f7695ecfea0f0c53f4c5cd52f1f705da378c7a33d13f1d102f" already present on machine 12m Normal AddedInterface pod/router-default-9c66fb948-4h58s Add eth0 [10.134.0.9/23] from ovn-kubernetes 7m36s Normal SuccessfulCreate replicaset/kube-auth-proxy-77df8579b6 Created pod: kube-auth-proxy-77df8579b6-fd2ss 7m36s Normal SuccessfulCreate replicaset/kube-auth-proxy-77df8579b6 Created pod: kube-auth-proxy-77df8579b6-57ph6 7m36s Warning FailedCreate replicaset/kube-auth-proxy-77df8579b6 Error creating: pods "kube-auth-proxy-77df8579b6-" is forbidden: error looking up service account openshift-ingress/kube-auth-proxy: serviceaccount "kube-auth-proxy" not found 7m36s Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-77df8579b6 from 0 to 2 7m35s Normal Pulling pod/kube-auth-proxy-77df8579b6-fd2ss Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 7m35s Normal AddedInterface pod/kube-auth-proxy-77df8579b6-fd2ss Add eth0 [10.134.0.27/23] from ovn-kubernetes 7m35s Normal AddedInterface pod/kube-auth-proxy-77df8579b6-57ph6 Add eth0 [10.133.0.20/23] from ovn-kubernetes 7m35s Normal Pulling pod/kube-auth-proxy-77df8579b6-57ph6 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" 7m34s Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7b7995d5cd Created pod: istiod-openshift-gateway-7b7995d5cd-52gjw 7m34s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7b7995d5cd from 0 to 1 7m32s Normal Pulled pod/kube-auth-proxy-77df8579b6-fd2ss Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.779s (2.779s including waiting). Image size: 265120132 bytes. 7m32s Normal Created pod/kube-auth-proxy-77df8579b6-57ph6 Created container: kube-auth-proxy 7m32s Normal Started pod/kube-auth-proxy-77df8579b6-57ph6 Started container kube-auth-proxy 7m32s Normal Pulled pod/kube-auth-proxy-77df8579b6-57ph6 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy:v3.5.0-ea.2" in 2.694s (2.694s including waiting). Image size: 265120132 bytes. 7m32s Normal Created pod/kube-auth-proxy-77df8579b6-fd2ss Created container: kube-auth-proxy 7m32s Normal Started pod/kube-auth-proxy-77df8579b6-fd2ss Started container kube-auth-proxy 7m32s Normal Pulling pod/istiod-openshift-gateway-7b7995d5cd-52gjw Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" 7m32s Normal AddedInterface pod/istiod-openshift-gateway-7b7995d5cd-52gjw Add eth0 [10.133.0.21/23] from ovn-kubernetes 7m30s Normal Pulled pod/istiod-openshift-gateway-7b7995d5cd-52gjw Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" in 2.663s (2.663s including waiting). Image size: 181126104 bytes. 7m29s Normal Started pod/istiod-openshift-gateway-7b7995d5cd-52gjw Started container discovery 7m29s Normal Created pod/istiod-openshift-gateway-7b7995d5cd-52gjw Created container: discovery 7m28s Warning ProbeError pod/istiod-openshift-gateway-7b7995d5cd-52gjw Readiness probe error: HTTP probe failed with statuscode: 503... 7m28s Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-6cbcdcfccc from 0 to 1 7m28s Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Add eth0 [10.132.0.31/23] from ovn-kubernetes 7m28s Normal Pulling pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" 7m28s Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-6cbcdcfccc Created pod: data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 7m28s Warning Unhealthy pod/istiod-openshift-gateway-7b7995d5cd-52gjw Readiness probe failed: HTTP probe failed with statuscode: 503 7m21s Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m20s Normal Pulled pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" in 7.791s (7.791s including waiting). Image size: 181318953 bytes. 7m20s Normal Created pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Created container: istio-proxy 7m20s Normal Started pod/data-science-gateway-data-science-gateway-class-6cbcdcfcccj7977 Started container istio-proxy 7m18s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-5677ccb49b from 0 to 1 7m18s Normal SuccessfulCreate replicaset/istiod-openshift-gateway-5677ccb49b Created pod: istiod-openshift-gateway-5677ccb49b-dr5l5 7m17s Normal AddedInterface pod/istiod-openshift-gateway-5677ccb49b-dr5l5 Add eth0 [10.133.0.22/23] from ovn-kubernetes 7m17s Normal Created pod/istiod-openshift-gateway-5677ccb49b-dr5l5 Created container: discovery 7m17s Normal Started pod/istiod-openshift-gateway-5677ccb49b-dr5l5 Started container discovery 7m17s Normal Pulled pod/istiod-openshift-gateway-5677ccb49b-dr5l5 Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31" already present on machine 7m16s Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7b7995d5cd Deleted pod: istiod-openshift-gateway-7b7995d5cd-52gjw 7m16s Normal Killing pod/istiod-openshift-gateway-7b7995d5cd-52gjw Stopping container discovery 7m16s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7b7995d5cd from 1 to 0 7m6s Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 7m6s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 7m4s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m4s Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 6m49s Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 6m49s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 5m5s Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 5m5s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-c747fcbf8 from 0 to 1 5m5s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-c747fcbf8 Created pod: maas-default-gateway-openshift-default-c747fcbf8-m7vfh 5m4s Normal AddedInterface pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Add eth0 [10.132.0.42/23] from ovn-kubernetes 3m47s Warning FailedCreate replicaset/payload-pre-processing-59fbc47bf8 Error creating: pods "payload-pre-processing-59fbc47bf8-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 3m47s Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-59fbc47bf8 from 0 to 1 3m47s Warning FailedCreate replicaset/payload-processing-5b54b7fbb7 Error creating: pods "payload-processing-5b54b7fbb7-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 3m47s Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5b54b7fbb7 from 0 to 1 3m46s Normal Pulling pod/payload-processing-5b54b7fbb7-pqcb8 Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 3m46s Normal Pulling pod/payload-pre-processing-59fbc47bf8-9v8gw Pulling image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" 3m46s Normal AddedInterface pod/payload-pre-processing-59fbc47bf8-9v8gw Add eth0 [10.134.0.31/23] from ovn-kubernetes 3m46s Normal AddedInterface pod/payload-processing-5b54b7fbb7-pqcb8 Add eth0 [10.133.0.32/23] from ovn-kubernetes 3m46s Normal SuccessfulCreate replicaset/payload-processing-5b54b7fbb7 Created pod: payload-processing-5b54b7fbb7-pqcb8 3m46s Normal SuccessfulCreate replicaset/payload-pre-processing-59fbc47bf8 Created pod: payload-pre-processing-59fbc47bf8-9v8gw 3m44s Normal Pulled pod/payload-processing-5b54b7fbb7-pqcb8 Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 1.34s (1.34s including waiting). Image size: 166453408 bytes. 3m44s Normal Created pod/payload-processing-5b54b7fbb7-pqcb8 Created container: payload-processing 3m44s Normal Started pod/payload-processing-5b54b7fbb7-pqcb8 Started container payload-processing 3m43s Normal Created pod/payload-pre-processing-59fbc47bf8-9v8gw Created container: payload-pre-processing 3m43s Normal Pulled pod/payload-pre-processing-59fbc47bf8-9v8gw Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" in 2.376s (2.376s including waiting). Image size: 166453408 bytes. 3m43s Normal Started pod/payload-pre-processing-59fbc47bf8-9v8gw Started container payload-pre-processing 3m36s Normal Pulled pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec" already present on machine 3m36s Normal Created pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Created container: istio-proxy 3m36s Normal Started pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Started container istio-proxy 3m31s Warning ProbeError pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Startup probe error: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused... 3m31s Warning Unhealthy pod/maas-default-gateway-openshift-default-c747fcbf8-m7vfh Startup probe failed: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused 37s Normal Killing pod/payload-pre-processing-59fbc47bf8-9v8gw Stopping container payload-pre-processing 37s Normal Killing pod/payload-processing-5b54b7fbb7-pqcb8 Stopping container payload-processing 35s Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-5b54b7fbb7 from 0 to 1 35s Normal SuccessfulCreate replicaset/payload-processing-5b54b7fbb7 Created pod: payload-processing-5b54b7fbb7-l487m 35s Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-59fbc47bf8 from 0 to 1 35s Normal SuccessfulCreate replicaset/payload-pre-processing-59fbc47bf8 Created pod: payload-pre-processing-59fbc47bf8-z5v5b 34s Normal Created pod/payload-processing-5b54b7fbb7-l487m Created container: payload-processing 34s Normal Pulled pod/payload-pre-processing-59fbc47bf8-z5v5b Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 34s Normal AddedInterface pod/payload-pre-processing-59fbc47bf8-z5v5b Add eth0 [10.133.0.37/23] from ovn-kubernetes 34s Normal Created pod/payload-pre-processing-59fbc47bf8-z5v5b Created container: payload-pre-processing 34s Normal Started pod/payload-pre-processing-59fbc47bf8-z5v5b Started container payload-pre-processing 34s Normal AddedInterface pod/payload-processing-5b54b7fbb7-l487m Add eth0 [10.133.0.36/23] from ovn-kubernetes 34s Normal Pulled pod/payload-processing-5b54b7fbb7-l487m Container image "quay.io/opendatahub/odh-ai-gateway-payload-processing:odh-stable" already present on machine 34s Normal Started pod/payload-processing-5b54b7fbb7-l487m Started container payload-processing