=== events in openshift-ingress (2026-10-31T18:45:52Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 12m Normal Scheduled pod/kube-auth-proxy-6d79b9f777-njg2g Successfully assigned openshift-ingress/kube-auth-proxy-6d79b9f777-njg2g to ip-10-0-136-243.us-west-2.compute.internal 11m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb25w to ip-10-0-140-53.us-west-2.compute.internal 8m1s Normal Scheduled pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-td8tp to ip-10-0-140-53.us-west-2.compute.internal 8m1s Normal Scheduled pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vhxhf to ip-10-0-136-243.us-west-2.compute.internal 7m46s Normal Scheduled pod/payload-processing-55fbbbf9dc-hzd7p Successfully assigned openshift-ingress/payload-processing-55fbbbf9dc-hzd7p to ip-10-0-140-53.us-west-2.compute.internal 8m31s Normal Scheduled pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-ftpvl to ip-10-0-140-53.us-west-2.compute.internal 19m Normal Scheduled pod/router-default-85b9df4f49-ndfrx Successfully assigned openshift-ingress/router-default-85b9df4f49-ndfrx to ip-10-0-141-104.us-west-2.compute.internal 12m Normal Scheduled pod/kube-auth-proxy-6d79b9f777-rs279 Successfully assigned openshift-ingress/kube-auth-proxy-6d79b9f777-rs279 to ip-10-0-140-53.us-west-2.compute.internal 19m Warning FailedScheduling pod/router-default-85b9df4f49-ndfrx no nodes available to schedule pods 26m Warning FailedScheduling pod/router-default-85b9df4f49-ndfrx no nodes available to schedule pods 13m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc to ip-10-0-141-104.us-west-2.compute.internal 7m46s Normal Scheduled pod/payload-pre-processing-7b98bb678c-xqczb Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-xqczb to ip-10-0-140-53.us-west-2.compute.internal 12m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 to ip-10-0-141-104.us-west-2.compute.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rmvck to ip-10-0-141-104.us-west-2.compute.internal 13m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-6rxqf Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-6rxqf to ip-10-0-140-53.us-west-2.compute.internal 8m31s Normal Scheduled pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 to ip-10-0-136-243.us-west-2.compute.internal 26m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-85b9df4f49 from 0 to 1 26m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 26m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 26m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 26m Normal SuccessfulCreate replicaset/router-default-85b9df4f49 Created pod: router-default-85b9df4f49-ndfrx 19m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 19m Warning FailedMount pod/router-default-85b9df4f49-ndfrx MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 19m Warning FailedMount pod/router-default-85b9df4f49-ndfrx MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 19m Warning FailedMount pod/router-default-85b9df4f49-ndfrx MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 19m Warning FailedMount pod/router-default-85b9df4f49-ndfrx MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 19m Normal AddedInterface pod/router-default-85b9df4f49-ndfrx Add eth0 [10.132.0.18/23] from ovn-kubernetes 18m Warning ProbeError pod/router-default-85b9df4f49-ndfrx Startup probe error: HTTP probe failed with statuscode: 500... 18m Normal Pulled pod/router-default-85b9df4f49-ndfrx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 18m Normal Created pod/router-default-85b9df4f49-ndfrx Container created 18m Normal Started pod/router-default-85b9df4f49-ndfrx Container started 18m Warning Unhealthy pod/router-default-85b9df4f49-ndfrx Startup probe failed: HTTP probe failed with statuscode: 500 13m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 13m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-6rxqf 13m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 13m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-6rxqf Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 13m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-6rxqf Add eth0 [10.134.0.21/23] from ovn-kubernetes 13m Normal Started pod/istiod-openshift-gateway-7876bfcb74-6rxqf Container started 13m Normal Created pod/istiod-openshift-gateway-7876bfcb74-6rxqf Container created 13m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-6rxqf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.727s (4.727s including waiting). Image size: 185476477 bytes. 13m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-6rxqf Readiness probe failed: HTTP probe failed with statuscode: 503 13m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-6rxqf Readiness probe error: HTTP probe failed with statuscode: 503... 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-5df48fcc99 from 0 to 1 13m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-599fb7fb95 from 0 to 1 13m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Add eth0 [10.132.0.33/23] from ovn-kubernetes 13m Normal Pulling pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-5df48fcc99 Created pod: maas-default-gateway-openshift-default-5df48fcc99-rmvck 13m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Add eth0 [10.132.0.34/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-599fb7fb95 Created pod: openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc 13m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 13m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 13m Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.967s (9.967s including waiting). Image size: 183555333 bytes. 13m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.951s (10.951s including waiting). Image size: 183555333 bytes. 13m Normal Started pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Container started 13m Normal Created pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Container created 13m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 13m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-56c7694f85 from 0 to 1 12m Normal SuccessfulCreate replicaset/kube-auth-proxy-6d79b9f777 Created pod: kube-auth-proxy-6d79b9f777-rs279 12m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-6d79b9f777 from 0 to 2 12m Normal AddedInterface pod/kube-auth-proxy-6d79b9f777-njg2g Add eth0 [10.133.0.26/23] from ovn-kubernetes 12m Normal Pulling pod/kube-auth-proxy-6d79b9f777-njg2g Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 12m Normal Pulling pod/kube-auth-proxy-6d79b9f777-rs279 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 12m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-56c7694f85 Created pod: data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 12m Normal AddedInterface pod/kube-auth-proxy-6d79b9f777-rs279 Add eth0 [10.134.0.24/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/kube-auth-proxy-6d79b9f777 Created pod: kube-auth-proxy-6d79b9f777-njg2g 12m Normal Pulled pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Add eth0 [10.132.0.40/23] from ovn-kubernetes 12m Normal Created pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Container created 12m Normal Started pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Container started 12m Normal Created pod/kube-auth-proxy-6d79b9f777-njg2g Container created 12m Normal Pulled pod/kube-auth-proxy-6d79b9f777-njg2g Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.116s (3.116s including waiting). Image size: 188904677 bytes. 12m Normal Pulled pod/kube-auth-proxy-6d79b9f777-rs279 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.2s (3.2s including waiting). Image size: 188904677 bytes. 12m Normal Started pod/kube-auth-proxy-6d79b9f777-rs279 Container started 12m Normal Started pod/kube-auth-proxy-6d79b9f777-njg2g Container started 12m Normal Created pod/kube-auth-proxy-6d79b9f777-rs279 Container created 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-wb25w 11m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 11m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Container started 11m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Add eth0 [10.134.0.26/23] from ovn-kubernetes 11m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 11m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Container created 11m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Readiness probe failed: HTTP probe failed with statuscode: 503 11m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-wb25w Readiness probe error: HTTP probe failed with statuscode: 503... 11m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-6rxqf 11m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 11m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-6rxqf Stopping container discovery 10m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Normal Started pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Container started 10m Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Normal Created pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Container created 10m Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Startup probe error: Get "http://10.132.0.33:15021/healthz/ready": dial tcp 10.132.0.33:15021: connect: connection refused... 10m Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Startup probe failed: Get "http://10.132.0.33:15021/healthz/ready": dial tcp 10.132.0.33:15021: connect: connection refused 9m48s Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m48s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m32s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-5df48fcc99 from 1 to 3 8m32s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 8m32s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-5df48fcc99 Created pod: maas-default-gateway-openshift-default-5df48fcc99-ftpvl 8m32s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-5df48fcc99 Created pod: maas-default-gateway-openshift-default-5df48fcc99-b9cb8 8m31s Normal AddedInterface pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Add eth0 [10.133.0.31/23] from ovn-kubernetes 8m31s Normal Pulling pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 8m31s Normal Pulling pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 8m31s Normal AddedInterface pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Add eth0 [10.134.0.36/23] from ovn-kubernetes 8m27s Normal Created pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Container created 8m27s Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.259s (4.259s including waiting). Image size: 183555333 bytes. 8m27s Normal Created pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Container created 8m27s Normal Started pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Container started 8m27s Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.185s (4.185s including waiting). Image size: 183555333 bytes. 8m27s Normal Started pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Container started 8m21s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused... 8m21s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused 8m21s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": dial tcp 10.133.0.31:15021: connect: connection refused 8m21s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused... 8m19s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Startup probe failed: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 8m19s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Startup probe error: Get "http://10.133.0.31:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 8m19s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 8m19s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-ftpvl Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 8m2s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-5df48fcc99 Created pod: maas-default-gateway-openshift-default-5df48fcc99-vhxhf 8m2s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-5df48fcc99 Created pod: maas-default-gateway-openshift-default-5df48fcc99-td8tp 8m2s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-5df48fcc99 from 3 to 5 8m2s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 8m1s Normal AddedInterface pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Add eth0 [10.134.0.38/23] from ovn-kubernetes 8m1s Normal AddedInterface pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Add eth0 [10.133.0.34/23] from ovn-kubernetes 8m1s Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m1s Normal Created pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Container created 8m1s Normal Started pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Container started 8m1s Normal Started pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Container started 8m1s Normal Pulled pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m1s Normal Created pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Container created 7m54s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused 7m54s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused... 7m54s Warning ProbeError pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 7m54s Warning Unhealthy pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 7m47s Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 7m47s Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-55fbbbf9dc from 0 to 1 7m47s Warning FailedCreate replicaset/payload-processing-55fbbbf9dc Error creating: pods "payload-processing-55fbbbf9dc-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 7m47s Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 7m46s Normal AddedInterface pod/payload-pre-processing-7b98bb678c-xqczb Add eth0 [10.134.0.40/23] from ovn-kubernetes 7m46s Normal Pulling pod/payload-pre-processing-7b98bb678c-xqczb Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 7m46s Normal SuccessfulCreate replicaset/payload-processing-55fbbbf9dc Created pod: payload-processing-55fbbbf9dc-hzd7p 7m46s Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-xqczb 7m45s Normal AddedInterface pod/payload-processing-55fbbbf9dc-hzd7p Add eth0 [10.134.0.41/23] from ovn-kubernetes 7m45s Normal Pulling pod/payload-processing-55fbbbf9dc-hzd7p Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 7m42s Normal Started pod/payload-processing-55fbbbf9dc-hzd7p Container started 7m42s Normal Pulled pod/payload-processing-55fbbbf9dc-hzd7p Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.782s (2.782s including waiting). Image size: 125353963 bytes. 7m42s Normal Pulled pod/payload-pre-processing-7b98bb678c-xqczb Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.23s (3.23s including waiting). Image size: 125353963 bytes. 7m42s Normal Started pod/payload-pre-processing-7b98bb678c-xqczb Container started 7m42s Normal Created pod/payload-pre-processing-7b98bb678c-xqczb Container created 7m42s Normal Created pod/payload-processing-55fbbbf9dc-hzd7p Container created 7m26s Warning NodeNotReady pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Node is not ready 7m26s Warning NodeNotReady pod/router-default-85b9df4f49-ndfrx Node is not ready 7m26s Warning NodeNotReady pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Node is not ready 7m26s Warning NodeNotReady pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Node is not ready 7m20s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m20s Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m15s Normal TaintManagerEviction pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Cancelling deletion of Pod openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rmvck 7m15s Normal TaintManagerEviction pod/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 Cancelling deletion of Pod openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85hfgj7 7m15s Normal TaintManagerEviction pod/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc Cancelling deletion of Pod openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-5j9qc 7m15s Normal TaintManagerEviction pod/router-default-85b9df4f49-ndfrx Cancelling deletion of Pod openshift-ingress/router-default-85b9df4f49-ndfrx 7m1s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m1s Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 2m30s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 2m30s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-5df48fcc99 Deleted pod: maas-default-gateway-openshift-default-5df48fcc99-td8tp 2m30s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-5df48fcc99 from 5 to 3 2m30s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-5df48fcc99 Deleted pod: maas-default-gateway-openshift-default-5df48fcc99-vhxhf 2m29s Normal Killing pod/maas-default-gateway-openshift-default-5df48fcc99-td8tp Stopping container istio-proxy 2m29s Normal Killing pod/maas-default-gateway-openshift-default-5df48fcc99-vhxhf Stopping container istio-proxy 2m14s Normal Killing pod/maas-default-gateway-openshift-default-5df48fcc99-rmvck Stopping container istio-proxy 2m14s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-5df48fcc99 Deleted pod: maas-default-gateway-openshift-default-5df48fcc99-rmvck 2m14s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-5df48fcc99 Deleted pod: maas-default-gateway-openshift-default-5df48fcc99-b9cb8 2m14s Normal Killing pod/maas-default-gateway-openshift-default-5df48fcc99-b9cb8 Stopping container istio-proxy 2m14s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 1; reason: All metrics below target 2m14s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-5df48fcc99 from 3 to 1