=== events in openshift-ingress (2026-11-04T02:42:53Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 18m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mrstj to ip-10-0-141-221.us-west-2.compute.internal 6m32s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 to ip-10-0-141-221.us-west-2.compute.internal 7m34s Normal Scheduled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Successfully assigned openshift-ingress/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm to ip-10-0-141-221.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Successfully assigned openshift-ingress/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 to ip-10-0-137-185.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Successfully assigned openshift-ingress/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x to ip-10-0-129-205.us-west-2.compute.internal 8m2s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv to ip-10-0-137-185.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Successfully assigned openshift-ingress/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 to ip-10-0-129-205.us-west-2.compute.internal 22m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-74vwf Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-74vwf to ip-10-0-141-221.us-west-2.compute.internal 7m18s Normal Scheduled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Successfully assigned openshift-ingress/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq to ip-10-0-141-221.us-west-2.compute.internal 15m Normal Scheduled pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zqvb2 to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Successfully assigned openshift-ingress/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q to ip-10-0-137-185.us-west-2.compute.internal 6m32s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk to ip-10-0-141-221.us-west-2.compute.internal 20m Normal Scheduled pod/kube-auth-proxy-b56c7b57b-kvxjl Successfully assigned openshift-ingress/kube-auth-proxy-b56c7b57b-kvxjl to ip-10-0-129-205.us-west-2.compute.internal 6m33s Normal Scheduled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Successfully assigned openshift-ingress/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv to ip-10-0-129-205.us-west-2.compute.internal 5m1s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 to ip-10-0-129-205.us-west-2.compute.internal 6m17s Normal Scheduled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Successfully assigned openshift-ingress/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 to ip-10-0-137-185.us-west-2.compute.internal 15m Normal Scheduled pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vtg49 to ip-10-0-129-205.us-west-2.compute.internal 15m Normal Scheduled pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-d22h9 to ip-10-0-141-221.us-west-2.compute.internal 7m44s Normal Scheduled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Successfully assigned openshift-ingress/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj to ip-10-0-129-205.us-west-2.compute.internal 22m Normal Scheduled pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-c89j6 to ip-10-0-137-185.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Successfully assigned openshift-ingress/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Successfully assigned openshift-ingress/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l to ip-10-0-129-205.us-west-2.compute.internal 15m Normal Scheduled pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6wkwd to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 to ip-10-0-129-205.us-west-2.compute.internal 28m Normal Scheduled pod/router-default-5657c749cf-m5w29 Successfully assigned openshift-ingress/router-default-5657c749cf-m5w29 to ip-10-0-141-221.us-west-2.compute.internal 22m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg to ip-10-0-137-185.us-west-2.compute.internal 20m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb to ip-10-0-137-185.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 to ip-10-0-141-221.us-west-2.compute.internal 8m44s Normal Scheduled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Successfully assigned openshift-ingress/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt to ip-10-0-141-221.us-west-2.compute.internal 35m Warning FailedScheduling pod/router-default-5657c749cf-m5w29 no nodes available to schedule pods 13m Normal Scheduled pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g to ip-10-0-141-221.us-west-2.compute.internal 11m Normal Scheduled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Successfully assigned openshift-ingress/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz to ip-10-0-137-185.us-west-2.compute.internal 13m Normal Scheduled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Successfully assigned openshift-ingress/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg to ip-10-0-141-221.us-west-2.compute.internal 15m Normal Scheduled pod/payload-pre-processing-7b98bb678c-fjz9p Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-fjz9p to ip-10-0-129-205.us-west-2.compute.internal 5m1s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf to ip-10-0-129-205.us-west-2.compute.internal 5m1s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr to ip-10-0-137-185.us-west-2.compute.internal 20m Normal Scheduled pod/kube-auth-proxy-b56c7b57b-zsj9v Successfully assigned openshift-ingress/kube-auth-proxy-b56c7b57b-zsj9v to ip-10-0-141-221.us-west-2.compute.internal 28m Warning FailedScheduling pod/router-default-5657c749cf-m5w29 no nodes available to schedule pods 15m Normal Scheduled pod/payload-processing-7bb9489676-zb9d2 Successfully assigned openshift-ingress/payload-processing-7bb9489676-zb9d2 to ip-10-0-141-221.us-west-2.compute.internal 5m1s Normal Scheduled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Successfully assigned openshift-ingress/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt to ip-10-0-141-221.us-west-2.compute.internal 6m33s Normal Scheduled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Successfully assigned openshift-ingress/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z to ip-10-0-129-205.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 to ip-10-0-141-221.us-west-2.compute.internal 13m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf to ip-10-0-141-221.us-west-2.compute.internal 35m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-5657c749cf from 0 to 1 35m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 35m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 35m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 35m Normal SuccessfulCreate replicaset/router-default-5657c749cf Created pod: router-default-5657c749cf-m5w29 28m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 28m Warning FailedMount pod/router-default-5657c749cf-m5w29 MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 28m Warning FailedMount pod/router-default-5657c749cf-m5w29 MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 27m Normal Pulled pod/router-default-5657c749cf-m5w29 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 27m Normal Started pod/router-default-5657c749cf-m5w29 Container started 27m Normal Created pod/router-default-5657c749cf-m5w29 Container created 27m Normal AddedInterface pod/router-default-5657c749cf-m5w29 Add eth0 [10.132.0.11/23] from ovn-kubernetes 27m Warning ProbeError pod/router-default-5657c749cf-m5w29 Startup probe error: HTTP probe failed with statuscode: 500... 27m Warning Unhealthy pod/router-default-5657c749cf-m5w29 Startup probe failed: HTTP probe failed with statuscode: 500 27m Normal TaintManagerEviction pod/router-default-5657c749cf-m5w29 Cancelling deletion of Pod openshift-ingress/router-default-5657c749cf-m5w29 22m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 22m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 22m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-74vwf 22m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-74vwf Add eth0 [10.132.0.26/23] from ovn-kubernetes 22m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-74vwf Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 22m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-74vwf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.917s (3.917s including waiting). Image size: 185476477 bytes. 22m Normal Created pod/istiod-openshift-gateway-7876bfcb74-74vwf Container created 22m Normal Started pod/istiod-openshift-gateway-7876bfcb74-74vwf Container started 22m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-668d64f647 Created pod: openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg 22m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-668d64f647 from 0 to 1 22m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 22m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7df854bf67 from 0 to 1 22m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7df854bf67 Created pod: maas-default-gateway-openshift-default-7df854bf67-c89j6 22m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-74vwf Readiness probe error: HTTP probe failed with statuscode: 503... 22m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-74vwf Readiness probe failed: HTTP probe failed with statuscode: 503 22m Normal AddedInterface pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Add eth0 [10.134.0.28/23] from ovn-kubernetes 22m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 22m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Add eth0 [10.134.0.29/23] from ovn-kubernetes 22m Normal Pulling pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 22m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 22m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 12.293s (12.293s including waiting). Image size: 183555333 bytes. 22m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 12.647s (12.647s including waiting). Image size: 183555333 bytes. 22m Normal Created pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Container created 22m Normal Started pod/openshift-ai-inference-openshift-ai-inference-668d64f647-xzdhg Container started 22m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 22m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 21m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 21m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 20m Normal SuccessfulCreate replicaset/kube-auth-proxy-b56c7b57b Created pod: kube-auth-proxy-b56c7b57b-kvxjl 20m Normal Pulling pod/kube-auth-proxy-b56c7b57b-zsj9v Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 20m Normal Pulling pod/kube-auth-proxy-b56c7b57b-kvxjl Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 20m Normal AddedInterface pod/kube-auth-proxy-b56c7b57b-kvxjl Add eth0 [10.133.0.25/23] from ovn-kubernetes 20m Normal AddedInterface pod/kube-auth-proxy-b56c7b57b-zsj9v Add eth0 [10.132.0.28/23] from ovn-kubernetes 20m Normal SuccessfulCreate replicaset/kube-auth-proxy-b56c7b57b Created pod: kube-auth-proxy-b56c7b57b-zsj9v 20m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-575956cc9c from 0 to 1 20m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-575956cc9c Created pod: data-science-gateway-data-science-gateway-class-575956cc9cxg4wb 20m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-b56c7b57b from 0 to 2 20m Normal Started pod/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb Container started 20m Normal Created pod/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb Container created 20m Normal Pulled pod/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 20m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-575956cc9cxg4wb Add eth0 [10.134.0.42/23] from ovn-kubernetes 20m Normal Started pod/kube-auth-proxy-b56c7b57b-zsj9v Container started 20m Normal Created pod/kube-auth-proxy-b56c7b57b-zsj9v Container created 20m Normal Pulled pod/kube-auth-proxy-b56c7b57b-zsj9v Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.17s (3.17s including waiting). Image size: 188904677 bytes. 20m Normal Pulled pod/kube-auth-proxy-b56c7b57b-kvxjl Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.149s (3.149s including waiting). Image size: 188904677 bytes. 20m Normal Started pod/kube-auth-proxy-b56c7b57b-kvxjl Container started 20m Normal Created pod/kube-auth-proxy-b56c7b57b-kvxjl Container created 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 19m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 19m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 18m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 18m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-mrstj 18m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Container started 18m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Container created 18m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 18m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Add eth0 [10.132.0.31/23] from ovn-kubernetes 18m Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Readiness probe error: HTTP probe failed with statuscode: 503... 18m Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-mrstj Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-74vwf Stopping container discovery 18m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 18m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-74vwf 18m Normal Started pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Container started 18m Normal Created pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Container created 18m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused 18m Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Startup probe error: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused... 18m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 15m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7df854bf67 Created pod: maas-default-gateway-openshift-default-7df854bf67-vtg49 15m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7df854bf67 from 1 to 4 15m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7df854bf67 Created pod: maas-default-gateway-openshift-default-7df854bf67-d22h9 15m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7df854bf67 Created pod: maas-default-gateway-openshift-default-7df854bf67-6wkwd 15m Normal AddedInterface pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Add eth0 [10.133.0.32/23] from ovn-kubernetes 15m Normal Pulling pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 15m Normal AddedInterface pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Add eth0 [10.132.0.39/23] from ovn-kubernetes 15m Normal Pulling pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 15m Normal Pulling pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 15m Normal AddedInterface pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Add eth0 [10.133.0.33/23] from ovn-kubernetes 15m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 5.941s (5.941s including waiting). Image size: 183555333 bytes. 15m Normal Started pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Container started 15m Normal Started pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Container started 15m Normal Created pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Container created 15m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 6.085s (6.085s including waiting). Image size: 183555333 bytes. 15m Normal Started pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Container started 15m Normal Created pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Container created 15m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 5.598s (5.598s including waiting). Image size: 183555333 bytes. 15m Normal Created pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Container created 15m Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Startup probe failed: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused 15m Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Startup probe error: Get "http://10.133.0.32:15021/healthz/ready": dial tcp 10.133.0.32:15021: connect: connection refused... 15m Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused 15m Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Startup probe error: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused... 15m Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Startup probe failed: Get "http://10.132.0.39:15021/healthz/ready": dial tcp 10.132.0.39:15021: connect: connection refused 15m Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-d22h9 Startup probe error: Get "http://10.132.0.39:15021/healthz/ready": dial tcp 10.132.0.39:15021: connect: connection refused... 15m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7df854bf67 from 4 to 5 15m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7df854bf67 Created pod: maas-default-gateway-openshift-default-7df854bf67-zqvb2 15m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 15m Normal AddedInterface pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Add eth0 [10.132.0.41/23] from ovn-kubernetes 15m Normal Pulled pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 15m Normal Started pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Container started 15m Normal Created pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Container created 15m Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Startup probe failed: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused 15m Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Startup probe error: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused... 15m Warning FailedCreate replicaset/payload-processing-7bb9489676 Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 15m Normal SuccessfulCreate replicaset/payload-processing-7bb9489676 Created pod: payload-processing-7bb9489676-zb9d2 15m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 15m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 15m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-7bb9489676 from 0 to 1 15m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-fjz9p 15m Normal AddedInterface pod/payload-processing-7bb9489676-zb9d2 Add eth0 [10.132.0.42/23] from ovn-kubernetes 15m Normal Pulling pod/payload-processing-7bb9489676-zb9d2 Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 15m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-fjz9p Add eth0 [10.133.0.35/23] from ovn-kubernetes 15m Normal Pulling pod/payload-pre-processing-7b98bb678c-fjz9p Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 15m Normal Started pod/payload-pre-processing-7b98bb678c-fjz9p Container started 15m Normal Started pod/payload-processing-7bb9489676-zb9d2 Container started 15m Normal Created pod/payload-processing-7bb9489676-zb9d2 Container created 15m Normal Pulled pod/payload-pre-processing-7b98bb678c-fjz9p Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.19s (3.19s including waiting). Image size: 125353963 bytes. 15m Normal Created pod/payload-pre-processing-7b98bb678c-fjz9p Container created 15m Normal Pulled pod/payload-processing-7bb9489676-zb9d2 Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.035s (3.035s including waiting). Image size: 125353963 bytes. 13m Normal AddedLabel gateway/e2e-worker-w5-10dba4 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-10dba4 13m Normal AddedLabel gateway/e2e-worker-w1-019d33 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-019d33 13m Normal ScalingReplicaSet deployment/e2e-worker-w5-10dba4-openshift-default Scaled up replica set e2e-worker-w5-10dba4-openshift-default-845669679f from 0 to 1 13m Normal SuccessfulCreate replicaset/e2e-worker-w5-10dba4-openshift-default-845669679f Created pod: e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x 13m Normal SuccessfulCreate replicaset/e2e-worker-w1-019d33-openshift-default-595cc78699 Created pod: e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg 13m Normal AddedLabel gateway/e2e-worker-w2-b8a6a7 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-b8a6a7 13m Normal AddedLabel gateway/e2e-worker-w3-e981f0 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-e981f0 13m Normal ScalingReplicaSet deployment/e2e-worker-w1-019d33-openshift-default Scaled up replica set e2e-worker-w1-019d33-openshift-default-595cc78699 from 0 to 1 13m Normal AddedLabel gateway/e2e-worker-w4-7cdaa3 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-7cdaa3 13m Normal ScalingReplicaSet deployment/e2e-worker-w3-e981f0-openshift-default Scaled up replica set e2e-worker-w3-e981f0-openshift-default-94bf498f from 0 to 1 13m Normal SuccessfulCreate replicaset/e2e-worker-w3-e981f0-openshift-default-94bf498f Created pod: e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w 13m Normal ScalingReplicaSet deployment/e2e-worker-w4-7cdaa3-openshift-default Scaled up replica set e2e-worker-w4-7cdaa3-openshift-default-86d488b67f from 0 to 1 13m Normal ScalingReplicaSet deployment/e2e-worker-w2-b8a6a7-openshift-default Scaled up replica set e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 from 0 to 1 13m Normal AddedInterface pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Add eth0 [10.133.0.37/23] from ovn-kubernetes 13m Normal AddedInterface pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Add eth0 [10.133.0.38/23] from ovn-kubernetes 13m Normal AddedInterface pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Add eth0 [10.132.0.43/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 13m Normal SuccessfulCreate replicaset/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 Created pod: e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l 13m Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Add eth0 [10.133.0.39/23] from ovn-kubernetes 13m Normal AddedInterface pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Add eth0 [10.132.0.44/23] from ovn-kubernetes 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-10dba4 Scaled up replica set payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7 from 0 to 1 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-10dba4 Scaled up replica set payload-processing-e2e-worker-w5-10dba4-856677df76 from 0 to 1 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7 Created pod: payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7 Error creating: pods "payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-10dba4: serviceaccount "payload-processing-e2e-worker-w5-10dba4" not found 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-10dba4-856677df76 Error creating: pods "payload-processing-e2e-worker-w5-10dba4-856677df76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-10dba4: serviceaccount "payload-processing-e2e-worker-w5-10dba4" not found 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Add eth0 [10.132.0.45/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-10dba4-856677df76 Created pod: payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt 13m Normal AddedInterface pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Add eth0 [10.132.0.46/23] from ovn-kubernetes 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Started pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Container started 13m Normal Created pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Container created 13m Normal Pulled pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Container created 13m Normal Started pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Container started 13m Normal SuccessfulCreate replicaset/e2e-worker-w6-d13b9f-openshift-default-77b577f959 Created pod: e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 13m Normal ScalingReplicaSet deployment/e2e-worker-w6-d13b9f-openshift-default Scaled up replica set e2e-worker-w6-d13b9f-openshift-default-77b577f959 from 0 to 1 13m Normal AddedLabel gateway/e2e-worker-w6-d13b9f Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-d13b9f 13m Normal AddedInterface pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Add eth0 [10.133.0.40/23] from ovn-kubernetes 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-019d33 Scaled up replica set payload-processing-e2e-worker-w1-019d33-774cc4b768 from 0 to 1 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-019d33-774cc4b768 Created pod: payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-019d33 Scaled up replica set payload-pre-processing-e2e-worker-w1-019d33-bf979c959 from 0 to 1 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-019d33-774cc4b768 Error creating: pods "payload-processing-e2e-worker-w1-019d33-774cc4b768-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-019d33: serviceaccount "payload-processing-e2e-worker-w1-019d33" not found 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-019d33-bf979c959 Error creating: pods "payload-pre-processing-e2e-worker-w1-019d33-bf979c959-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-019d33: serviceaccount "payload-processing-e2e-worker-w1-019d33" not found 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-019d33-bf979c959 Created pod: payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal AddedInterface pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Add eth0 [10.132.0.48/23] from ovn-kubernetes 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Add eth0 [10.133.0.41/23] from ovn-kubernetes 13m Normal Started pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Container started 13m Normal Pulled pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Started pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Container started 13m Normal Created pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Container created 13m Normal Created pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Container created 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-7cdaa3 Scaled up replica set payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5 from 0 to 1 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-7cdaa3 Scaled up replica set payload-processing-e2e-worker-w4-7cdaa3-59c679f476 from 0 to 1 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-7cdaa3-59c679f476 Error creating: pods "payload-processing-e2e-worker-w4-7cdaa3-59c679f476-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-7cdaa3: serviceaccount "payload-processing-e2e-worker-w4-7cdaa3" not found 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5 Created pod: payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-7cdaa3-59c679f476 Created pod: payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5 Error creating: pods "payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-7cdaa3: serviceaccount "payload-processing-e2e-worker-w4-7cdaa3" not found 13m Normal AddedInterface pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Add eth0 [10.132.0.49/23] from ovn-kubernetes 13m Normal Created pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Container created 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Container created 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Add eth0 [10.132.0.50/23] from ovn-kubernetes 13m Normal Pulled pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Started pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Container started 13m Normal Started pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Container started 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff Error creating: pods "payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-b8a6a7: serviceaccount "payload-processing-e2e-worker-w2-b8a6a7" not found 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-b8a6a7 Scaled up replica set payload-processing-e2e-worker-w2-b8a6a7-66b644b65b from 0 to 1 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b Error creating: pods "payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-b8a6a7: serviceaccount "payload-processing-e2e-worker-w2-b8a6a7" not found 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-b8a6a7 Scaled up replica set payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff from 0 to 1 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff Created pod: payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv 13m Normal Started pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Container started 13m Normal Pulled pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Container created 13m Normal SuccessfulCreate replicaset/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 Created pod: e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q 13m Normal Created pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Container created 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal AddedLabel gateway/e2e-worker-w0-796b7b Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-796b7b 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Add eth0 [10.133.0.43/23] from ovn-kubernetes 13m Normal ScalingReplicaSet deployment/e2e-worker-w0-796b7b-openshift-default Scaled up replica set e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 from 0 to 1 13m Normal AddedInterface pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Add eth0 [10.133.0.44/23] from ovn-kubernetes 13m Normal Started pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Container started 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b Created pod: payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 13m Normal AddedInterface pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Add eth0 [10.134.0.56/23] from ovn-kubernetes 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-e981f0 Scaled up replica set payload-processing-e2e-worker-w3-e981f0-6f497b7554 from 0 to 1 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-e981f0-6f497b7554 Error creating: pods "payload-processing-e2e-worker-w3-e981f0-6f497b7554-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-e981f0: serviceaccount "payload-processing-e2e-worker-w3-e981f0" not found 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-e981f0 Scaled up replica set payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc from 0 to 1 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc Error creating: pods "payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-e981f0: serviceaccount "payload-processing-e2e-worker-w3-e981f0" not found 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-e981f0-6f497b7554 Created pod: payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc Created pod: payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 13m Normal Created pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Container created 13m Normal Created pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Container created 13m Normal Started pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Container started 13m Normal AddedInterface pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Add eth0 [10.133.0.46/23] from ovn-kubernetes 13m Normal Pulled pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Started pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Container started 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Add eth0 [10.133.0.45/23] from ovn-kubernetes 13m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 13m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-d13b9f Scaled up replica set payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875 from 0 to 1 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875 Error creating: pods "payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-d13b9f: serviceaccount "payload-processing-e2e-worker-w6-d13b9f" not found 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-d13b9f-756674d988 Created pod: payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-d13b9f Scaled up replica set payload-processing-e2e-worker-w6-d13b9f-756674d988 from 0 to 1 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875 Created pod: payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-d13b9f-756674d988 Error creating: pods "payload-processing-e2e-worker-w6-d13b9f-756674d988-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-d13b9f: serviceaccount "payload-processing-e2e-worker-w6-d13b9f" not found 13m Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Container started 13m Normal Started pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Container started 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Add eth0 [10.132.0.53/23] from ovn-kubernetes 13m Normal Pulled pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal AddedInterface pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Add eth0 [10.133.0.47/23] from ovn-kubernetes 13m Normal Created pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Container created 13m Normal Created pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Container created 13m Normal Started pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Container started 13m Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Container created 13m Normal Started pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Container started 13m Warning ProbeError pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Readiness probe error: Get "http://10.132.0.43:15021/healthz/ready": read tcp 10.132.0.2:56506->10.132.0.43:15021: read: connection reset by peer... 13m Warning Unhealthy pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Readiness probe failed: Get "http://10.132.0.43:15021/healthz/ready": read tcp 10.132.0.2:56506->10.132.0.43:15021: read: connection reset by peer 13m Normal Created pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Container created 13m Normal Pulled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused 13m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f Error creating: pods "payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-796b7b: serviceaccount "payload-processing-e2e-worker-w0-796b7b" not found 13m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-796b7b-5494c86db8 Created pod: payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g 13m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-796b7b Scaled up replica set payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f from 0 to 1 13m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-796b7b Scaled up replica set payload-processing-e2e-worker-w0-796b7b-5494c86db8 from 0 to 1 13m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-796b7b-5494c86db8 Error creating: pods "payload-processing-e2e-worker-w0-796b7b-5494c86db8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-796b7b: serviceaccount "payload-processing-e2e-worker-w0-796b7b" not found 13m Warning ProbeError pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Startup probe error: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Startup probe failed: Get "http://10.132.0.43:15021/healthz/ready": dial tcp 10.132.0.43:15021: connect: connection refused 13m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f Created pod: payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm 13m Normal Created pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Container created 13m Normal Started pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Container started 13m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Add eth0 [10.132.0.54/23] from ovn-kubernetes 13m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Created pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Container created 13m Normal Started pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Container started 13m Normal AddedInterface pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Add eth0 [10.132.0.55/23] from ovn-kubernetes 13m Normal Pulled pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 13m Normal Pulled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Container created 13m Normal Started pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Container started 13m Warning Unhealthy pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Readiness probe failed: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 13m Warning ProbeError pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Readiness probe error: Get "http://10.133.0.37:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 13m Warning ProbeError pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Startup probe error: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused... 13m Warning Unhealthy pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Startup probe failed: Get "http://10.133.0.38:15021/healthz/ready": dial tcp 10.133.0.38:15021: connect: connection refused 12m Normal Created pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Container created 12m Normal Pulled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Container started 12m Normal Pulled pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Created pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Container created 12m Normal Started pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Container started 12m Warning Unhealthy pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Startup probe failed: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused 12m Warning ProbeError pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Startup probe error: Get "http://10.133.0.37:15021/healthz/ready": dial tcp 10.133.0.37:15021: connect: connection refused... 12m Warning Unhealthy pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Startup probe failed: Get "http://10.132.0.44:15021/healthz/ready": dial tcp 10.132.0.44:15021: connect: connection refused 12m Warning ProbeError pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Startup probe error: Get "http://10.132.0.44:15021/healthz/ready": dial tcp 10.132.0.44:15021: connect: connection refused... 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-e981f0-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-7cdaa3-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-7cdaa3-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-e981f0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Normal Created pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Container created 12m Normal Pulled pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Container started 12m Warning ProbeError pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 12m Warning Unhealthy pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-b8a6a7-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-b8a6a7-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-e981f0-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-e981f0-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-d13b9f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-d13b9f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Normal Created pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Container created 12m Normal Started pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Container started 12m Normal Pulled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Warning ProbeError pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Startup probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 12m Warning Unhealthy pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Startup probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-d13b9f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-d13b9f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Normal SuccessfulCreate replicaset/e2e-worker-w1-019d33-openshift-default-595cc78699 Created pod: e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz 11m Normal AddedInterface pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Add eth0 [10.134.0.65/23] from ovn-kubernetes 11m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 11m Normal Pulled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal ScalingReplicaSet deployment/e2e-worker-w1-019d33-openshift-default Scaled up replica set e2e-worker-w1-019d33-openshift-default-595cc78699 from 1 to 2 11m Normal Started pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Container started 11m Normal Created pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Container created 11m Warning Unhealthy pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 10m Normal SuccessfulCreate replicaset/e2e-worker-w5-10dba4-openshift-default-845669679f Created pod: e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 10m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 10m Normal ScalingReplicaSet deployment/e2e-worker-w5-10dba4-openshift-default Scaled up replica set e2e-worker-w5-10dba4-openshift-default-845669679f from 1 to 2 10m Normal Started pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Container started 10m Normal Created pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Container created 10m Normal AddedInterface pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Add eth0 [10.134.0.67/23] from ovn-kubernetes 10m Normal Pulled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m58s Warning Unhealthy pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 9m58s Warning ProbeError pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 9m50s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-7df854bf67 from 5 to 4 9m50s Normal Killing pod/maas-default-gateway-openshift-default-7df854bf67-6wkwd Stopping container istio-proxy 9m50s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-7df854bf67 Deleted pod: maas-default-gateway-openshift-default-7df854bf67-6wkwd 9m50s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 8m45s Normal ScalingReplicaSet deployment/e2e-worker-w0-796b7b-openshift-default Scaled up replica set e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 from 1 to 2 8m45s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 8m45s Normal SuccessfulCreate replicaset/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 Created pod: e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc 8m44s Normal Created pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Container created 8m44s Normal Started pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Container started 8m44s Normal Pulled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m44s Normal AddedInterface pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Add eth0 [10.132.0.61/23] from ovn-kubernetes 8m38s Warning Unhealthy pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused 8m38s Warning ProbeError pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Startup probe error: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused... 8m14s Normal Killing pod/payload-processing-e2e-worker-w3-e981f0-6f497b7554-6dsh5 Stopping container payload-processing 8m14s Normal Killing pod/payload-pre-processing-e2e-worker-w3-e981f0-dd76865bc-d7vp5 Stopping container payload-pre-processing 8m4s Normal Killing pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Stopping container payload-pre-processing 8m4s Normal Killing pod/maas-default-gateway-openshift-default-7df854bf67-zqvb2 Stopping container istio-proxy 8m4s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 8m4s Normal Killing pod/payload-processing-e2e-worker-w6-d13b9f-756674d988-5kgm5 Stopping container payload-processing 8m4s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-7df854bf67 Deleted pod: maas-default-gateway-openshift-default-7df854bf67-zqvb2 8m4s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-7df854bf67 from 4 to 3 8m3s Normal ScalingReplicaSet deployment/e2e-worker-w4-7cdaa3-openshift-default Scaled up replica set e2e-worker-w4-7cdaa3-openshift-default-86d488b67f from 1 to 2 8m3s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv 8m3s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-7cdaa3-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 8m2s Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m2s Normal Killing pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Stopping container payload-processing 8m2s Normal Killing pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Stopping container payload-pre-processing 8m2s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Add eth0 [10.134.0.77/23] from ovn-kubernetes 8m2s Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Container created 8m2s Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Container started 7m55s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 7m55s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 7m49s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-7df854bf67 from 3 to 2 7m49s Normal Killing pod/maas-default-gateway-openshift-default-7df854bf67-vtg49 Stopping container istio-proxy 7m49s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-7df854bf67 Deleted pod: maas-default-gateway-openshift-default-7df854bf67-vtg49 7m49s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 7m48s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-d13b9f-5578f97875-l25pt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m47s Normal Killing pod/e2e-worker-w3-e981f0-openshift-default-94bf498f-bv28w Stopping container istio-proxy 7m44s Normal ScalingReplicaSet deployment/e2e-worker-w0-796b7b-openshift-default Scaled up replica set e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 from 2 to 3 7m44s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-796b7b-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 7m44s Normal SuccessfulCreate replicaset/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5 Created pod: e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj 7m44s Normal AddedInterface pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Add eth0 [10.133.0.59/23] from ovn-kubernetes 7m43s Normal Created pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Container created 7m43s Normal Pulled pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m43s Normal Started pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Container started 7m41s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-10dba4-6c9cc644b7-n8bt8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m41s Warning Unhealthy pod/payload-processing-e2e-worker-w5-10dba4-856677df76-b8djt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 7m40s Normal Killing pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Stopping container istio-proxy 7m38s Warning ProbeError pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 7m38s Warning Unhealthy pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 7m36s Warning Unhealthy pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Readiness probe failed: HTTP probe failed with statuscode: 503 7m36s Warning ProbeError pod/e2e-worker-w6-d13b9f-openshift-default-77b577f959-6ngv2 Readiness probe error: HTTP probe failed with statuscode: 503... 7m34s Normal ScalingReplicaSet deployment/e2e-worker-w5-10dba4-openshift-default Scaled up replica set e2e-worker-w5-10dba4-openshift-default-845669679f from 2 to 3 7m34s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-10dba4-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 7m34s Normal SuccessfulCreate replicaset/e2e-worker-w5-10dba4-openshift-default-845669679f Created pod: e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm 7m33s Normal Started pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Container started 7m33s Normal AddedInterface pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Add eth0 [10.132.0.63/23] from ovn-kubernetes 7m33s Normal Pulled pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m33s Normal Created pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Container created 7m26s Warning ProbeError pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Startup probe error: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused... 7m26s Warning Unhealthy pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Startup probe failed: Get "http://10.132.0.63:15021/healthz/ready": dial tcp 10.132.0.63:15021: connect: connection refused 7m20s Normal Killing pod/e2e-worker-w5-10dba4-openshift-default-845669679f-59jwm Stopping container istio-proxy 7m20s Normal Killing pod/e2e-worker-w5-10dba4-openshift-default-845669679f-hn68x Stopping container istio-proxy 7m20s Normal Killing pod/e2e-worker-w5-10dba4-openshift-default-845669679f-bsvx7 Stopping container istio-proxy 7m18s Normal SuccessfulCreate replicaset/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 Created pod: e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq 7m18s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-b8a6a7-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 7m18s Normal ScalingReplicaSet deployment/e2e-worker-w2-b8a6a7-openshift-default Scaled up replica set e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 from 1 to 2 7m17s Normal AddedInterface pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Add eth0 [10.132.0.65/23] from ovn-kubernetes 7m17s Normal Pulled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m17s Normal Created pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Container created 7m17s Normal Started pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Container started 7m10s Warning ProbeError pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Startup probe error: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused... 7m10s Warning Unhealthy pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Startup probe failed: Get "http://10.132.0.65:15021/healthz/ready": dial tcp 10.132.0.65:15021: connect: connection refused 7m7s Normal Killing pod/payload-processing-e2e-worker-w0-796b7b-5494c86db8-65t8g Stopping container payload-processing 7m7s Normal Killing pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Stopping container payload-pre-processing 6m49s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-796b7b-56c8bfb57f-tpwxm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m33s Normal Killing pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-ch4gj Stopping container istio-proxy 6m33s Normal SuccessfulCreate replicaset/e2e-worker-w1-019d33-openshift-default-595cc78699 Created pod: e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv 6m33s Normal Killing pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Stopping container istio-proxy 6m33s Warning ProbeError pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Readiness probe error: HTTP probe failed with statuscode: 503... 6m33s Normal ScalingReplicaSet deployment/e2e-worker-w1-019d33-openshift-default Scaled up replica set e2e-worker-w1-019d33-openshift-default-595cc78699 from 2 to 4 6m33s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-019d33-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 6m33s Warning Unhealthy pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-2fp9q Readiness probe failed: HTTP probe failed with statuscode: 503 6m33s Normal SuccessfulCreate replicaset/e2e-worker-w1-019d33-openshift-default-595cc78699 Created pod: e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z 6m33s Normal Killing pod/e2e-worker-w0-796b7b-openshift-default-7d858dd5f5-m8kwc Stopping container istio-proxy 6m32s Normal AddedInterface pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Add eth0 [10.133.0.62/23] from ovn-kubernetes 6m32s Normal Pulled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m32s Normal Created pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Container created 6m32s Normal Started pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Container started 6m32s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 6m32s Normal Pulled pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m32s Normal AddedInterface pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Add eth0 [10.133.0.61/23] from ovn-kubernetes 6m32s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk 6m32s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-7cdaa3-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 6m32s Normal Created pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Container created 6m32s Normal ScalingReplicaSet deployment/e2e-worker-w4-7cdaa3-openshift-default Scaled up replica set e2e-worker-w4-7cdaa3-openshift-default-86d488b67f from 2 to 4 6m32s Normal Started pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Container started 6m31s Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Container created 6m31s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Add eth0 [10.132.0.68/23] from ovn-kubernetes 6m31s Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m31s Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Container started 6m31s Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Container started 6m31s Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Container created 6m31s Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m31s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Add eth0 [10.132.0.67/23] from ovn-kubernetes 6m25s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 6m25s Warning Unhealthy pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 6m25s Warning ProbeError pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 6m25s Warning Unhealthy pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 6m25s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 6m25s Warning ProbeError pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 6m25s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 6m25s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 6m19s Normal Killing pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Stopping container payload-pre-processing 6m19s Normal Killing pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Stopping container payload-processing 6m18s Normal ScalingReplicaSet deployment/e2e-worker-w2-b8a6a7-openshift-default Scaled up replica set e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 from 2 to 3 6m18s Normal SuccessfulCreate replicaset/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59 Created pod: e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 6m18s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-b8a6a7-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 6m17s Normal Started pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Container started 6m17s Normal Created pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Container created 6m17s Normal Pulled pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m17s Normal AddedInterface pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Add eth0 [10.134.0.82/23] from ovn-kubernetes 6m10s Warning ProbeError pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Startup probe error: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused... 6m10s Warning Unhealthy pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Startup probe failed: Get "http://10.134.0.82:15021/healthz/ready": dial tcp 10.134.0.82:15021: connect: connection refused 5m56s Warning Unhealthy pod/payload-processing-e2e-worker-w1-019d33-774cc4b768-wn6gf Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m56s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-019d33-bf979c959-s5ql8 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m53s Normal Killing pod/payload-processing-e2e-worker-w2-b8a6a7-66b644b65b-px6t7 Stopping container payload-processing 5m53s Normal Killing pod/e2e-worker-w1-019d33-openshift-default-595cc78699-h4nnv Stopping container istio-proxy 5m53s Normal Killing pod/payload-pre-processing-e2e-worker-w2-b8a6a7-7599bd99ff-nhmsv Stopping container payload-pre-processing 5m53s Normal Killing pod/e2e-worker-w1-019d33-openshift-default-595cc78699-gp77z Stopping container istio-proxy 5m53s Normal Killing pod/e2e-worker-w1-019d33-openshift-default-595cc78699-vkmgz Stopping container istio-proxy 5m53s Normal Killing pod/e2e-worker-w1-019d33-openshift-default-595cc78699-s4rzg Stopping container istio-proxy 5m27s Normal Killing pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Stopping container istio-proxy 5m27s Normal Killing pod/payload-processing-e2e-worker-w4-7cdaa3-59c679f476-l6wm9 Stopping container payload-processing 5m27s Normal Killing pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Stopping container payload-pre-processing 5m27s Normal Killing pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Stopping container istio-proxy 5m27s Normal Killing pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-8b44l Stopping container istio-proxy 5m24s Warning ProbeError pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Readiness probe error: HTTP probe failed with statuscode: 503... 5m24s Warning Unhealthy pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Readiness probe failed: HTTP probe failed with statuscode: 503 5m24s Warning ProbeError pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-9qljq Readiness probe error: HTTP probe failed with statuscode: 503... 5m24s Warning Unhealthy pod/e2e-worker-w2-b8a6a7-openshift-default-598fc69c59-wrwf7 Readiness probe failed: HTTP probe failed with statuscode: 503 5m3s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-7cdaa3-6db5b4b7c5-r89hf Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m1s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf 5m1s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm 5m1s Normal ScalingReplicaSet deployment/e2e-worker-w4-7cdaa3-openshift-default Scaled up replica set e2e-worker-w4-7cdaa3-openshift-default-86d488b67f from 4 to 8 5m1s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-7cdaa3-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 5m1s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Add eth0 [10.133.0.63/23] from ovn-kubernetes 5m1s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Add eth0 [10.132.0.70/23] from ovn-kubernetes 5m1s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Add eth0 [10.133.0.64/23] from ovn-kubernetes 5m1s Normal AddedInterface pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Add eth0 [10.134.0.86/23] from ovn-kubernetes 5m1s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s 5m1s Normal SuccessfulCreate replicaset/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f Created pod: e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr 5m Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Container started 5m Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Container started 5m Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Container started 5m Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Container created 5m Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Container created 5m Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Container created 5m Normal Started pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Container started 5m Normal Created pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Container created 5m Normal Pulled pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-zmgmf Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-qbtt5 Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-dhcxr Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-fvn8s Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pmzrm Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Stopping container istio-proxy 4m55s Normal Killing pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Stopping container istio-proxy 4m54s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Readiness probe error: HTTP probe failed with statuscode: 503... 4m54s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-htzxk Readiness probe failed: HTTP probe failed with statuscode: 503 4m54s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Readiness probe error: HTTP probe failed with statuscode: 503... 4m54s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Readiness probe failed: HTTP probe failed with statuscode: 503 4m54s Warning ProbeError pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-wbfsv Readiness probe error: HTTP probe failed with statuscode: 503... 4m54s Warning Unhealthy pod/e2e-worker-w4-7cdaa3-openshift-default-86d488b67f-pstf7 Readiness probe failed: HTTP probe failed with statuscode: 503 2m31s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-7df854bf67 from 2 to 1 2m31s Normal Killing pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Stopping container istio-proxy 2m31s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-7df854bf67 Deleted pod: maas-default-gateway-openshift-default-7df854bf67-c89j6 2m31s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 1; reason: All metrics below target 2m30s Warning ProbeError pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Readiness probe error: HTTP probe failed with statuscode: 503... 2m30s Warning Unhealthy pod/maas-default-gateway-openshift-default-7df854bf67-c89j6 Readiness probe failed: HTTP probe failed with statuscode: 503