=== events in openshift-ingress (2026-11-04T05:10:42Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 11m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr to ip-10-0-133-180.us-west-1.compute.internal 4m5s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn to ip-10-0-135-45.us-west-1.compute.internal 4m50s Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl to ip-10-0-135-45.us-west-1.compute.internal 19m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 to ip-10-0-135-45.us-west-1.compute.internal 5m36s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg to ip-10-0-135-45.us-west-1.compute.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9tvb6 to ip-10-0-141-132.us-west-1.compute.internal 4m5s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 to ip-10-0-141-132.us-west-1.compute.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t to ip-10-0-133-180.us-west-1.compute.internal 4m21s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 to ip-10-0-133-180.us-west-1.compute.internal 8m13s Normal Scheduled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Successfully assigned openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 to ip-10-0-135-45.us-west-1.compute.internal 12m Normal Scheduled pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 to ip-10-0-133-180.us-west-1.compute.internal 11m Normal Scheduled pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 to ip-10-0-141-132.us-west-1.compute.internal 7m22s Normal Scheduled pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Successfully assigned openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm to ip-10-0-135-45.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 to ip-10-0-133-180.us-west-1.compute.internal 7m7s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq to ip-10-0-135-45.us-west-1.compute.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vnmc7 to ip-10-0-141-132.us-west-1.compute.internal 18m Normal Scheduled pod/kube-auth-proxy-546d78d5c-vbwrt Successfully assigned openshift-ingress/kube-auth-proxy-546d78d5c-vbwrt to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Successfully assigned openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 to ip-10-0-141-132.us-west-1.compute.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4sj9h to ip-10-0-133-180.us-west-1.compute.internal 5m5s Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 to ip-10-0-141-132.us-west-1.compute.internal 9m37s Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 to ip-10-0-135-45.us-west-1.compute.internal 11m Normal Scheduled pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw to ip-10-0-133-180.us-west-1.compute.internal 11m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Successfully assigned openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-hp82n to ip-10-0-141-132.us-west-1.compute.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc to ip-10-0-141-132.us-west-1.compute.internal 23m Normal Scheduled pod/router-default-74447c8475-62kjf Successfully assigned openshift-ingress/router-default-74447c8475-62kjf to ip-10-0-135-45.us-west-1.compute.internal 24m Warning FailedScheduling pod/router-default-74447c8475-62kjf no nodes available to schedule pods 7m12s Normal Scheduled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Successfully assigned openshift-ingress/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 to ip-10-0-133-180.us-west-1.compute.internal 30m Warning FailedScheduling pod/router-default-74447c8475-62kjf no nodes available to schedule pods 5m35s Normal Scheduled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Successfully assigned openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw to ip-10-0-133-180.us-west-1.compute.internal 4m21s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d to ip-10-0-141-132.us-west-1.compute.internal 11m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj to ip-10-0-133-180.us-west-1.compute.internal 6m5s Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 to ip-10-0-141-132.us-west-1.compute.internal 11m Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp to ip-10-0-141-132.us-west-1.compute.internal 13m Normal Scheduled pod/payload-processing-7bb9489676-xvj6h Successfully assigned openshift-ingress/payload-processing-7bb9489676-xvj6h to ip-10-0-141-132.us-west-1.compute.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bnxlc to ip-10-0-141-132.us-west-1.compute.internal 7m51s Normal Scheduled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Successfully assigned openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 to ip-10-0-141-132.us-west-1.compute.internal 4m20s Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 to ip-10-0-133-180.us-west-1.compute.internal 19m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-g8rd9 to ip-10-0-133-180.us-west-1.compute.internal 4m20s Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd to ip-10-0-141-132.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw to ip-10-0-133-180.us-west-1.compute.internal 6m5s Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz to ip-10-0-133-180.us-west-1.compute.internal 6m5s Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq to ip-10-0-141-132.us-west-1.compute.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-d8rj2 to ip-10-0-133-180.us-west-1.compute.internal 4m21s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 to ip-10-0-141-132.us-west-1.compute.internal 14m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-j8ns4 to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Successfully assigned openshift-ingress/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm to ip-10-0-141-132.us-west-1.compute.internal 12m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv to ip-10-0-133-180.us-west-1.compute.internal 11m Normal Scheduled pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Successfully assigned openshift-ingress/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt to ip-10-0-135-45.us-west-1.compute.internal 12m Normal Scheduled pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh to ip-10-0-133-180.us-west-1.compute.internal 18m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr to ip-10-0-135-45.us-west-1.compute.internal 9m22s Normal Scheduled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Successfully assigned openshift-ingress/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh to ip-10-0-135-45.us-west-1.compute.internal 12m Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 to ip-10-0-135-45.us-west-1.compute.internal 13m Normal Scheduled pod/payload-pre-processing-7b98bb678c-n9sxt Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-n9sxt to ip-10-0-133-180.us-west-1.compute.internal 4m21s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g to ip-10-0-133-180.us-west-1.compute.internal 13m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-rl9f9 to ip-10-0-135-45.us-west-1.compute.internal 19m Normal Scheduled pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-tl7xs to ip-10-0-135-45.us-west-1.compute.internal 18m Normal Scheduled pod/kube-auth-proxy-546d78d5c-g8s92 Successfully assigned openshift-ingress/kube-auth-proxy-546d78d5c-g8s92 to ip-10-0-141-132.us-west-1.compute.internal 5m36s Normal Scheduled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Successfully assigned openshift-ingress/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 to ip-10-0-141-132.us-west-1.compute.internal 16m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-f447m Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f447m to ip-10-0-133-180.us-west-1.compute.internal 8m8s Normal Scheduled pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Successfully assigned openshift-ingress/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx to ip-10-0-133-180.us-west-1.compute.internal 12m Normal Scheduled pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw to ip-10-0-133-180.us-west-1.compute.internal 11m Normal Scheduled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Successfully assigned openshift-ingress/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm to ip-10-0-133-180.us-west-1.compute.internal 30m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-74447c8475 from 0 to 1 30m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 30m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 30m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 30m Normal SuccessfulCreate replicaset/router-default-74447c8475 Created pod: router-default-74447c8475-62kjf 24m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 23m Warning FailedMount pod/router-default-74447c8475-62kjf MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 23m Warning FailedMount pod/router-default-74447c8475-62kjf MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 23m Warning FailedMount pod/router-default-74447c8475-62kjf MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 23m Normal AddedInterface pod/router-default-74447c8475-62kjf Add eth0 [10.132.0.19/23] from ovn-kubernetes 23m Normal Pulled pod/router-default-74447c8475-62kjf Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 23m Normal Created pod/router-default-74447c8475-62kjf Container created 23m Normal Started pod/router-default-74447c8475-62kjf Container started 19m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 19m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-g8rd9 19m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 19m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 19m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Add eth0 [10.133.0.20/23] from ovn-kubernetes 19m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 4.11s (4.11s including waiting). Image size: 185476477 bytes. 19m Normal Started pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Container started 19m Normal Created pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Container created 19m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 0 to 1 19m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Readiness probe failed: HTTP probe failed with statuscode: 503 19m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Readiness probe error: HTTP probe failed with statuscode: 503... 19m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-tl7xs 19m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Add eth0 [10.132.0.36/23] from ovn-kubernetes 19m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-55f56bc75f Created pod: openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 19m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-55f56bc75f from 0 to 1 19m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 19m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 19m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Add eth0 [10.132.0.35/23] from ovn-kubernetes 19m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 19m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 19m Normal Started pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Container started 19m Normal Created pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Container created 19m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-55f56bc75f-qlxv8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.239s (9.239s including waiting). Image size: 183555333 bytes. 19m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.498s (9.498s including waiting). Image size: 183555333 bytes. 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 19m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 19m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-7845f94dd5 Created pod: data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr 18m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-7845f94dd5 from 0 to 1 18m Normal SuccessfulCreate replicaset/kube-auth-proxy-546d78d5c Created pod: kube-auth-proxy-546d78d5c-g8s92 18m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-546d78d5c from 0 to 2 18m Normal SuccessfulCreate replicaset/kube-auth-proxy-546d78d5c Created pod: kube-auth-proxy-546d78d5c-vbwrt 18m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr Add eth0 [10.132.0.45/23] from ovn-kubernetes 18m Normal Pulling pod/kube-auth-proxy-546d78d5c-vbwrt Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 18m Normal AddedInterface pod/kube-auth-proxy-546d78d5c-vbwrt Add eth0 [10.133.0.22/23] from ovn-kubernetes 18m Normal Pulling pod/kube-auth-proxy-546d78d5c-g8s92 Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 18m Normal Pulled pod/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 18m Normal Created pod/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr Container created 18m Normal Started pod/data-science-gateway-data-science-gateway-class-7845f94dd5gvmbr Container started 18m Normal AddedInterface pod/kube-auth-proxy-546d78d5c-g8s92 Add eth0 [10.134.0.28/23] from ovn-kubernetes 18m Normal Started pod/kube-auth-proxy-546d78d5c-vbwrt Container started 18m Normal Started pod/kube-auth-proxy-546d78d5c-g8s92 Container started 18m Normal Created pod/kube-auth-proxy-546d78d5c-vbwrt Container created 18m Normal Created pod/kube-auth-proxy-546d78d5c-g8s92 Container created 18m Normal Pulled pod/kube-auth-proxy-546d78d5c-g8s92 Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.882s (2.882s including waiting). Image size: 188904677 bytes. 18m Normal Pulled pod/kube-auth-proxy-546d78d5c-vbwrt Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.238s (3.238s including waiting). Image size: 188904677 bytes. 18m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 18m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 16m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-f447m Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 16m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-f447m Add eth0 [10.133.0.26/23] from ovn-kubernetes 16m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-f447m Container created 16m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 16m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-f447m Container started 16m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-f447m 16m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-g8rd9 16m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 16m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-g8rd9 Stopping container discovery 16m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 16m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Container created 16m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Container started 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Startup probe error: Get "http://10.132.0.35:15021/healthz/ready": dial tcp 10.132.0.35:15021: connect: connection refused... 16m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-tl7xs Startup probe failed: Get "http://10.132.0.35:15021/healthz/ready": dial tcp 10.132.0.35:15021: connect: connection refused 15m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-bnxlc 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-j8ns4 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Add eth0 [10.134.0.37/23] from ovn-kubernetes 14m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 14m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 1 to 4 14m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-4sj9h 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Add eth0 [10.133.0.34/23] from ovn-kubernetes 14m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 14m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Add eth0 [10.133.0.35/23] from ovn-kubernetes 14m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 14m Normal Pulling pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 14m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.974s (3.974s including waiting). Image size: 183555333 bytes. 14m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Container started 14m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.806s (3.806s including waiting). Image size: 183555333 bytes. 14m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.983s (3.983s including waiting). Image size: 183555333 bytes. 14m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Container created 14m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Container created 14m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Container created 14m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Container started 14m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Container started 14m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused 14m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused... 14m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused... 14m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Startup probe failed: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused 14m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused 14m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-4sj9h Startup probe error: Get "http://10.133.0.35:15021/healthz/ready": dial tcp 10.133.0.35:15021: connect: connection refused... 13m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Add eth0 [10.134.0.39/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-9tvb6 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 4 to 5 13m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Container created 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 13m Warning FailedCreate replicaset/payload-processing-7bb9489676 Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 13m Normal SuccessfulCreate replicaset/payload-processing-7bb9489676 Created pod: payload-processing-7bb9489676-xvj6h 13m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 13m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-n9sxt 13m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 13m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-7bb9489676 from 0 to 1 13m Normal Pulling pod/payload-pre-processing-7b98bb678c-n9sxt Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 13m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-n9sxt Add eth0 [10.133.0.37/23] from ovn-kubernetes 13m Normal Pulling pod/payload-processing-7bb9489676-xvj6h Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 13m Normal AddedInterface pod/payload-processing-7bb9489676-xvj6h Add eth0 [10.134.0.40/23] from ovn-kubernetes 13m Normal Started pod/payload-processing-7bb9489676-xvj6h Container started 13m Normal Created pod/payload-processing-7bb9489676-xvj6h Container created 13m Normal Pulled pod/payload-processing-7bb9489676-xvj6h Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.947s (2.947s including waiting). Image size: 125353963 bytes. 13m Normal Started pod/payload-pre-processing-7b98bb678c-n9sxt Container started 13m Normal Pulled pod/payload-pre-processing-7b98bb678c-n9sxt Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 2.988s (2.988s including waiting). Image size: 125353963 bytes. 13m Normal Created pod/payload-pre-processing-7b98bb678c-n9sxt Container created 13m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-6886b96f9 from 5 to 8 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Add eth0 [10.132.0.56/23] from ovn-kubernetes 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-rl9f9 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-vnmc7 13m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-6886b96f9 Created pod: maas-default-gateway-openshift-default-6886b96f9-d8rj2 13m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 13m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Container created 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Add eth0 [10.134.0.41/23] from ovn-kubernetes 13m Normal AddedInterface pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Add eth0 [10.133.0.39/23] from ovn-kubernetes 13m Normal Pulled pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 13m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Container created 13m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Container started 13m Normal Started pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Container started 13m Normal Created pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Container created 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Startup probe failed: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Startup probe error: Get "http://10.132.0.56:15021/healthz/ready": dial tcp 10.132.0.56:15021: connect: connection refused... 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused 13m Warning Unhealthy pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused... 13m Warning ProbeError pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 12m Normal AddedLabel gateway/e2e-worker-w4-3eb31c Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-3eb31c 12m Normal AddedLabel gateway/e2e-worker-w5-4cce33 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-4cce33 12m Normal SuccessfulCreate replicaset/e2e-worker-w2-370078-openshift-default-584c45775 Created pod: e2e-worker-w2-370078-openshift-default-584c45775-hp82n 12m Normal AddedLabel gateway/e2e-worker-w3-dab70d Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-dab70d 12m Normal ScalingReplicaSet deployment/e2e-worker-w2-370078-openshift-default Scaled up replica set e2e-worker-w2-370078-openshift-default-584c45775 from 0 to 1 12m Normal AddedLabel gateway/e2e-worker-w2-370078 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-370078 12m Normal AddedLabel gateway/e2e-worker-w1-1874d8 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-1874d8 12m Normal AddedLabel gateway/e2e-worker-w6-cedfca Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-cedfca 12m Normal ScalingReplicaSet deployment/e2e-worker-w4-3eb31c-openshift-default Scaled up replica set e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 from 0 to 1 12m Normal ScalingReplicaSet deployment/e2e-worker-w5-4cce33-openshift-default Scaled up replica set e2e-worker-w5-4cce33-openshift-default-747768654f from 0 to 1 12m Normal ScalingReplicaSet deployment/e2e-worker-w1-1874d8-openshift-default Scaled up replica set e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9 from 0 to 1 12m Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Add eth0 [10.133.0.40/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 12m Normal ScalingReplicaSet deployment/e2e-worker-w6-cedfca-openshift-default Scaled up replica set e2e-worker-w6-cedfca-openshift-default-6b99d7696f from 0 to 1 12m Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 12m Normal AddedInterface pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Add eth0 [10.134.0.42/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9 Created pod: e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm 12m Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 12m Normal ScalingReplicaSet deployment/e2e-worker-w3-dab70d-openshift-default Scaled up replica set e2e-worker-w3-dab70d-openshift-default-6496f5466c from 0 to 1 12m Normal AddedInterface pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Add eth0 [10.132.0.58/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/e2e-worker-w3-dab70d-openshift-default-6496f5466c Created pod: e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt 12m Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Add eth0 [10.133.0.41/23] from ovn-kubernetes 12m Normal AddedInterface pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Add eth0 [10.134.0.43/23] from ovn-kubernetes 12m Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Add eth0 [10.132.0.57/23] from ovn-kubernetes 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-4cce33 Scaled up replica set payload-processing-e2e-worker-w5-4cce33-56666d7678 from 0 to 1 12m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-4cce33-56666d7678 Error creating: pods "payload-processing-e2e-worker-w5-4cce33-56666d7678-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-4cce33: serviceaccount "payload-processing-e2e-worker-w5-4cce33" not found 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f Error creating: pods "payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-4cce33: serviceaccount "payload-processing-e2e-worker-w5-4cce33" not found 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f Created pod: payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-4cce33-56666d7678 Created pod: payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-4cce33 Scaled up replica set payload-pre-processing-e2e-worker-w5-4cce33-66b786457f from 0 to 1 12m Normal Created pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Container created 12m Normal AddedInterface pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Add eth0 [10.133.0.42/23] from ovn-kubernetes 12m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal SuccessfulCreate replicaset/e2e-worker-w0-0aae2f-openshift-default-5f99b6799 Created pod: e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 12m Normal ScalingReplicaSet deployment/e2e-worker-w0-0aae2f-openshift-default Scaled up replica set e2e-worker-w0-0aae2f-openshift-default-5f99b6799 from 0 to 1 12m Normal Started pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Container started 12m Normal Created pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Container created 12m Normal AddedLabel gateway/e2e-worker-w0-0aae2f Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-0aae2f 12m Normal Pulled pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal Started pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Container started 12m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Add eth0 [10.134.0.44/23] from ovn-kubernetes 12m Normal AddedInterface pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Add eth0 [10.134.0.45/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8 Created pod: payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw 12m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8 Error creating: pods "payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-3eb31c: serviceaccount "payload-processing-e2e-worker-w4-3eb31c" not found 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87 Error creating: pods "payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-3eb31c: serviceaccount "payload-processing-e2e-worker-w4-3eb31c" not found 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87 Created pod: payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-3eb31c Scaled up replica set payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87 from 0 to 1 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-3eb31c Scaled up replica set payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8 from 0 to 1 12m Normal Pulled pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal Created pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Container created 12m Normal AddedInterface pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Add eth0 [10.133.0.44/23] from ovn-kubernetes 12m Normal Started pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Container started 12m Normal Started pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Container started 12m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Add eth0 [10.133.0.43/23] from ovn-kubernetes 12m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal Created pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Container created 12m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d Error creating: pods "payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-cedfca: serviceaccount "payload-processing-e2e-worker-w6-cedfca" not found 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-cedfca Scaled up replica set payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d from 0 to 1 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-cedfca Scaled up replica set payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659 from 0 to 1 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659 Error creating: pods "payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-cedfca: serviceaccount "payload-processing-e2e-worker-w6-cedfca" not found 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d Created pod: payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Normal Started pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Container started 12m Normal Started pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Container started 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Normal Created pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Container created 12m Normal Created pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Container created 12m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Add eth0 [10.133.0.45/23] from ovn-kubernetes 12m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal Pulled pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659 Created pod: payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t 12m Normal AddedInterface pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Add eth0 [10.133.0.46/23] from ovn-kubernetes 12m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f Created pod: payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz 12m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f Error creating: pods "payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-370078: serviceaccount "payload-processing-e2e-worker-w2-370078" not found 12m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-370078 Scaled up replica set payload-processing-e2e-worker-w2-370078-dc9467d44 from 0 to 1 12m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-370078-dc9467d44 Error creating: pods "payload-processing-e2e-worker-w2-370078-dc9467d44-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-370078: serviceaccount "payload-processing-e2e-worker-w2-370078" not found 12m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-370078 Scaled up replica set payload-pre-processing-e2e-worker-w2-370078-6c55687f5f from 0 to 1 12m Normal Started pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Container started 12m Normal Created pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Container created 12m Normal Started pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Container started 12m Normal Created pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Container created 12m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Add eth0 [10.133.0.47/23] from ovn-kubernetes 12m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-370078-dc9467d44 Created pod: payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 12m Normal Pulled pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Add eth0 [10.133.0.48/23] from ovn-kubernetes 12m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 12m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-0aae2f-74664f8f76 Error creating: pods "payload-processing-e2e-worker-w0-0aae2f-74664f8f76-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-0aae2f: serviceaccount "payload-processing-e2e-worker-w0-0aae2f" not found 11m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c Error creating: pods "payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-0aae2f: serviceaccount "payload-processing-e2e-worker-w0-0aae2f" not found 11m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-0aae2f Scaled up replica set payload-processing-e2e-worker-w0-0aae2f-74664f8f76 from 0 to 1 11m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-0aae2f Scaled up replica set payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c from 0 to 1 11m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-0aae2f-74664f8f76 Created pod: payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v 11m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Add eth0 [10.133.0.49/23] from ovn-kubernetes 11m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal Created pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Container created 11m Normal Started pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Container started 11m Normal Pulled pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c Created pod: payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr 11m Normal AddedInterface pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Add eth0 [10.133.0.50/23] from ovn-kubernetes 11m Normal Started pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Container started 11m Normal Created pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Container created 11m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-1874d8 Scaled up replica set payload-processing-e2e-worker-w1-1874d8-649ccc674d from 0 to 1 11m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-1874d8-649ccc674d Error creating: pods "payload-processing-e2e-worker-w1-1874d8-649ccc674d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-1874d8: serviceaccount "payload-processing-e2e-worker-w1-1874d8" not found 11m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7 Error creating: pods "payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-1874d8: serviceaccount "payload-processing-e2e-worker-w1-1874d8" not found 11m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7 Created pod: payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj 11m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-1874d8 Scaled up replica set payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7 from 0 to 1 11m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-1874d8-649ccc674d Created pod: payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 11m Normal AddedInterface pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Add eth0 [10.134.0.49/23] from ovn-kubernetes 11m Normal Pulled pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal Started pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Container started 11m Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Add eth0 [10.134.0.50/23] from ovn-kubernetes 11m Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 11m Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp 11m Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm 11m Normal Created pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Container created 11m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Add eth0 [10.133.0.51/23] from ovn-kubernetes 11m Normal ScalingReplicaSet deployment/e2e-worker-w4-3eb31c-openshift-default Scaled up replica set e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 from 1 to 3 11m Normal Started pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Container started 11m Normal Created pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Container created 11m Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Container started 11m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc Error creating: pods "payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-dab70d: serviceaccount "payload-processing-e2e-worker-w3-dab70d" not found 11m Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Add eth0 [10.133.0.52/23] from ovn-kubernetes 11m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-dab70d Scaled up replica set payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc from 0 to 1 11m Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Container created 11m Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Container started 11m Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Container created 11m Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-dab70d-6ddd49994d Error creating: pods "payload-processing-e2e-worker-w3-dab70d-6ddd49994d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-dab70d: serviceaccount "payload-processing-e2e-worker-w3-dab70d" not found 11m Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Readiness probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Readiness probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused 11m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-dab70d Scaled up replica set payload-processing-e2e-worker-w3-dab70d-6ddd49994d from 0 to 1 11m Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Readiness probe failed: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Readiness probe error: Get "http://10.133.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Container created 11m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc Created pod: payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz 11m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-dab70d-6ddd49994d Created pod: payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw 11m Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Container started 11m Normal Pulled pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Add eth0 [10.133.0.53/23] from ovn-kubernetes 11m Normal Started pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Container started 11m Normal Created pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Container created 11m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 11m Normal Started pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Container started 11m Normal Created pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Container created 11m Normal AddedInterface pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Add eth0 [10.133.0.54/23] from ovn-kubernetes 11m Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Container started 11m Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Container created 11m Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Startup probe failed: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Startup probe error: Get "http://10.133.0.52:15021/healthz/ready": dial tcp 10.133.0.52:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused 11m Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused... 11m Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Container created 11m Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Container started 11m Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Normal Started pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Container started 11m Normal Created pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Container created 11m Normal Pulled pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Warning Unhealthy pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Startup probe failed: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Startup probe error: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused... 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Normal Created pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Container created 11m Normal Started pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Container started 11m Normal Pulled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Warning Unhealthy pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Startup probe error: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused... 11m Normal Created pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Container created 11m Normal Started pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Container started 11m Normal Started pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Container started 11m Normal Pulled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Pulled pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Created pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Container created 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 11m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-1874d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-1874d8-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 11m Warning ProbeError pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Startup probe error: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused... 11m Warning Unhealthy pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Startup probe failed: Get "http://10.132.0.58:15021/healthz/ready": dial tcp 10.132.0.58:15021: connect: connection refused 11m Warning Unhealthy pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Startup probe failed: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused 11m Warning ProbeError pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Startup probe error: Get "http://10.134.0.43:15021/healthz/ready": dial tcp 10.134.0.43:15021: connect: connection refused... 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-1874d8-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-1874d8-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m37s Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 9m37s Normal ScalingReplicaSet deployment/e2e-worker-w4-3eb31c-openshift-default Scaled up replica set e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 from 3 to 4 9m37s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 9m36s Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Add eth0 [10.132.0.68/23] from ovn-kubernetes 9m36s Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m36s Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Container created 9m36s Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Container started 9m31s Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 9m31s Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 9m22s Normal ScalingReplicaSet deployment/e2e-worker-w6-cedfca-openshift-default Scaled up replica set e2e-worker-w6-cedfca-openshift-default-6b99d7696f from 1 to 2 9m22s Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh 9m22s Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m22s Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Add eth0 [10.132.0.69/23] from ovn-kubernetes 9m22s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 9m21s Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Container started 9m21s Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Container created 9m16s Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 9m16s Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 8m13s Normal SuccessfulCreate replicaset/e2e-worker-w0-0aae2f-openshift-default-5f99b6799 Created pod: e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 8m13s Normal ScalingReplicaSet deployment/e2e-worker-w0-0aae2f-openshift-default Scaled up replica set e2e-worker-w0-0aae2f-openshift-default-5f99b6799 from 1 to 2 8m13s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 8m12s Normal Created pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Container created 8m12s Normal Pulled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m12s Normal AddedInterface pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Add eth0 [10.132.0.70/23] from ovn-kubernetes 8m12s Normal Started pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Container started 8m10s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-j8ns4 Stopping container istio-proxy 8m10s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 8 to 6 8m10s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: All metrics below target 8m10s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-vnmc7 Stopping container istio-proxy 8m10s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-j8ns4 8m10s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-vnmc7 8m8s Normal SuccessfulCreate replicaset/e2e-worker-w2-370078-openshift-default-584c45775 Created pod: e2e-worker-w2-370078-openshift-default-584c45775-h6rwx 8m8s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 8m8s Normal ScalingReplicaSet deployment/e2e-worker-w2-370078-openshift-default Scaled up replica set e2e-worker-w2-370078-openshift-default-584c45775 from 1 to 2 8m7s Normal AddedInterface pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Add eth0 [10.133.0.66/23] from ovn-kubernetes 8m7s Normal Started pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Container started 8m7s Normal Created pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Container created 8m7s Normal Pulled pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m5s Warning ProbeError pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 8m5s Warning Unhealthy pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 8m1s Warning Unhealthy pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Startup probe failed: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused 8m1s Warning ProbeError pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Startup probe error: Get "http://10.133.0.66:15021/healthz/ready": dial tcp 10.133.0.66:15021: connect: connection refused... 7m51s Normal SuccessfulCreate replicaset/e2e-worker-w3-dab70d-openshift-default-6496f5466c Created pod: e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 7m51s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 7m51s Normal ScalingReplicaSet deployment/e2e-worker-w3-dab70d-openshift-default Scaled up replica set e2e-worker-w3-dab70d-openshift-default-6496f5466c from 1 to 2 7m50s Normal AddedInterface pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Add eth0 [10.134.0.58/23] from ovn-kubernetes 7m50s Normal Pulled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m50s Normal Created pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Container created 7m50s Normal Started pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Container started 7m45s Warning ProbeError pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Startup probe error: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused... 7m45s Warning Unhealthy pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Startup probe failed: Get "http://10.134.0.58:15021/healthz/ready": dial tcp 10.134.0.58:15021: connect: connection refused 7m22s Normal AddedInterface pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Add eth0 [10.132.0.71/23] from ovn-kubernetes 7m22s Normal SuccessfulCreate replicaset/e2e-worker-w2-370078-openshift-default-584c45775 Created pod: e2e-worker-w2-370078-openshift-default-584c45775-rvgrm 7m22s Normal ScalingReplicaSet deployment/e2e-worker-w2-370078-openshift-default Scaled up replica set e2e-worker-w2-370078-openshift-default-584c45775 from 2 to 3 7m22s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-370078-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 7m22s Normal Pulled pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m21s Normal Created pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Container created 7m21s Normal Started pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Container started 7m16s Warning Unhealthy pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 7m16s Warning ProbeError pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 7m12s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-0aae2f-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 7m12s Normal ScalingReplicaSet deployment/e2e-worker-w0-0aae2f-openshift-default Scaled up replica set e2e-worker-w0-0aae2f-openshift-default-5f99b6799 from 2 to 3 7m12s Normal SuccessfulCreate replicaset/e2e-worker-w0-0aae2f-openshift-default-5f99b6799 Created pod: e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 7m12s Normal AddedInterface pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Add eth0 [10.133.0.69/23] from ovn-kubernetes 7m12s Normal Pulled pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m11s Normal Started pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Container started 7m11s Normal Created pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Container created 7m7s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq 7m7s Normal ScalingReplicaSet deployment/e2e-worker-w5-4cce33-openshift-default Scaled up replica set e2e-worker-w5-4cce33-openshift-default-747768654f from 1 to 2 7m7s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 7m6s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Container created 7m6s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m6s Warning ProbeError pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Startup probe error: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused... 7m6s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Add eth0 [10.132.0.73/23] from ovn-kubernetes 7m6s Warning Unhealthy pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Startup probe failed: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused 7m6s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Container started 7m Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Startup probe failed: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused 7m Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Startup probe error: Get "http://10.132.0.73:15021/healthz/ready": dial tcp 10.132.0.73:15021: connect: connection refused... 6m50s Normal Killing pod/payload-processing-e2e-worker-w0-0aae2f-74664f8f76-rzd5v Stopping container payload-processing 6m50s Normal Killing pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Stopping container payload-pre-processing 6m36s Normal Killing pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Stopping container payload-pre-processing 6m36s Normal Killing pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Stopping container payload-processing 6m34s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-0aae2f-5ff7778f8c-4tbhr Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m25s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 6 to 5 6m25s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 6m25s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-9tvb6 6m24s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-9tvb6 Stopping container istio-proxy 6m17s Normal Killing pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-x69j7 Stopping container istio-proxy 6m17s Normal Killing pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Stopping container istio-proxy 6m17s Normal Killing pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-r62d6 Stopping container istio-proxy 6m16s Warning Unhealthy pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Readiness probe failed: HTTP probe failed with statuscode: 503 6m16s Warning ProbeError pod/e2e-worker-w0-0aae2f-openshift-default-5f99b6799-wp285 Readiness probe error: HTTP probe failed with statuscode: 503... 6m12s Warning Unhealthy pod/payload-processing-e2e-worker-w1-1874d8-649ccc674d-57hd7 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m11s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-1874d8-78f4d9f8b7-9d2tj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 6m9s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 5 to 4 6m9s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 6m9s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-d8rj2 Stopping container istio-proxy 6m9s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-d8rj2 6m6s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-3eb31c-openshift-default New size: 7; reason: cpu resource utilization (percentage of request) above target 6m6s Normal ScalingReplicaSet deployment/e2e-worker-w4-3eb31c-openshift-default Scaled up replica set e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 from 4 to 7 6m5s Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m5s Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq 6m5s Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Container created 6m5s Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Container started 6m5s Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Container started 6m5s Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Add eth0 [10.134.0.61/23] from ovn-kubernetes 6m5s Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Add eth0 [10.134.0.62/23] from ovn-kubernetes 6m5s Normal AddedInterface pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Add eth0 [10.133.0.75/23] from ovn-kubernetes 6m5s Normal Started pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Container started 6m5s Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m5s Normal Pulled pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m5s Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Container created 6m5s Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm 6m5s Normal Created pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Container created 6m5s Normal SuccessfulCreate replicaset/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95 Created pod: e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 6m2s Normal Killing pod/payload-pre-processing-e2e-worker-w3-dab70d-695b6bf8dc-vlnmz Stopping container payload-pre-processing 6m2s Normal Killing pod/payload-processing-e2e-worker-w3-dab70d-6ddd49994d-ggggw Stopping container payload-processing 5m58s Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Startup probe failed: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused 5m58s Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Startup probe error: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused... 5m58s Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Startup probe error: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused... 5m58s Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Startup probe failed: Get "http://10.133.0.75:15021/healthz/ready": dial tcp 10.133.0.75:15021: connect: connection refused 5m58s Warning ProbeError pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Startup probe error: Get "http://10.134.0.62:15021/healthz/ready": dial tcp 10.134.0.62:15021: connect: connection refused... 5m58s Warning Unhealthy pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Startup probe failed: Get "http://10.134.0.61:15021/healthz/ready": dial tcp 10.134.0.61:15021: connect: connection refused 5m57s Normal Killing pod/e2e-worker-w1-1874d8-openshift-default-7d9c99f7c9-jl4fm Stopping container istio-proxy 5m54s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-rl9f9 5m54s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-rl9f9 Stopping container istio-proxy 5m54s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: All metrics below target 5m54s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 4 to 3 5m43s Normal Killing pod/payload-pre-processing-e2e-worker-w2-370078-6c55687f5f-gwlxz Stopping container payload-pre-processing 5m43s Normal Killing pod/payload-processing-e2e-worker-w2-370078-dc9467d44-l4644 Stopping container payload-processing 5m36s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg 5m36s Normal ScalingReplicaSet deployment/e2e-worker-w5-4cce33-openshift-default Scaled up replica set e2e-worker-w5-4cce33-openshift-default-747768654f from 2 to 4 5m36s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Add eth0 [10.134.0.63/23] from ovn-kubernetes 5m36s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 5m36s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 5m35s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Container created 5m35s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Container started 5m35s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m35s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Add eth0 [10.132.0.76/23] from ovn-kubernetes 5m35s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m35s Normal ScalingReplicaSet deployment/e2e-worker-w3-dab70d-openshift-default Scaled up replica set e2e-worker-w3-dab70d-openshift-default-6496f5466c from 2 to 3 5m35s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-dab70d-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 5m35s Normal SuccessfulCreate replicaset/e2e-worker-w3-dab70d-openshift-default-6496f5466c Created pod: e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw 5m35s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Container created 5m35s Normal Killing pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Stopping container payload-processing 5m35s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Container started 5m35s Normal Killing pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Stopping container payload-pre-processing 5m34s Normal AddedInterface pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Add eth0 [10.133.0.82/23] from ovn-kubernetes 5m34s Normal Pulled pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m34s Normal Created pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Container created 5m34s Normal Started pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Container started 5m29s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Startup probe error: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused... 5m29s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Startup probe failed: Get "http://10.134.0.63:15021/healthz/ready": dial tcp 10.134.0.63:15021: connect: connection refused 5m29s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Startup probe error: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused... 5m29s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Startup probe failed: Get "http://10.132.0.76:15021/healthz/ready": dial tcp 10.132.0.76:15021: connect: connection refused 5m29s Warning ProbeError pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Startup probe error: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused... 5m29s Warning Unhealthy pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Startup probe failed: Get "http://10.133.0.82:15021/healthz/ready": dial tcp 10.133.0.82:15021: connect: connection refused 5m22s Normal Killing pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Stopping container istio-proxy 5m22s Normal Killing pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-rllr2 Stopping container istio-proxy 5m22s Normal Killing pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-xczrw Stopping container istio-proxy 5m20s Warning Unhealthy pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Readiness probe failed: HTTP probe failed with statuscode: 503 5m20s Warning ProbeError pod/e2e-worker-w3-dab70d-openshift-default-6496f5466c-9hqpt Readiness probe error: HTTP probe failed with statuscode: 503... 5m19s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w4-3eb31c-5bc57d7f87-xfpgv Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m19s Warning Unhealthy pod/payload-processing-e2e-worker-w4-3eb31c-589ff8b4b8-v89hw Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 5m14s Normal Killing pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Stopping container istio-proxy 5m14s Normal Killing pod/e2e-worker-w2-370078-openshift-default-584c45775-rvgrm Stopping container istio-proxy 5m14s Normal Killing pod/e2e-worker-w2-370078-openshift-default-584c45775-h6rwx Stopping container istio-proxy 5m9s Warning Unhealthy pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Readiness probe failed: HTTP probe failed with statuscode: 503 5m9s Warning ProbeError pod/e2e-worker-w2-370078-openshift-default-584c45775-hp82n Readiness probe error: HTTP probe failed with statuscode: 503... 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-kc7b4 Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-rfzt9 Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-ctnfp Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-fgrfq Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-g4ctm Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-vpv56 Stopping container istio-proxy 5m7s Normal Killing pod/e2e-worker-w4-3eb31c-openshift-default-7f69d85f95-tc7tm Stopping container istio-proxy 5m5s Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 5m5s Normal ScalingReplicaSet deployment/e2e-worker-w6-cedfca-openshift-default Scaled up replica set e2e-worker-w6-cedfca-openshift-default-6b99d7696f from 2 to 3 5m5s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 5m4s Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Add eth0 [10.134.0.64/23] from ovn-kubernetes 5m4s Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 5m4s Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Container started 5m4s Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Container created 4m58s Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Startup probe error: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused... 4m58s Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Startup probe failed: Get "http://10.134.0.64:15021/healthz/ready": dial tcp 10.134.0.64:15021: connect: connection refused 4m50s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 4m50s Normal Killing pod/payload-processing-e2e-worker-w6-cedfca-c7c8f5b7d-hdqcw Stopping container payload-processing 4m50s Normal Killing pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Stopping container payload-pre-processing 4m50s Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl 4m50s Normal ScalingReplicaSet deployment/e2e-worker-w6-cedfca-openshift-default Scaled up replica set e2e-worker-w6-cedfca-openshift-default-6b99d7696f from 3 to 4 4m49s Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Add eth0 [10.132.0.79/23] from ovn-kubernetes 4m49s Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Container created 4m49s Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Container started 4m49s Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m42s Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused 4m42s Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": dial tcp 10.132.0.79:15021: connect: connection refused... 4m40s Normal Killing pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Stopping container payload-pre-processing 4m40s Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Startup probe error: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 4m40s Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Startup probe failed: Get "http://10.132.0.79:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 4m40s Normal Killing pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Stopping container payload-processing 4m27s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-cedfca-67b95cc659-6v86t Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m23s Warning Unhealthy pod/payload-processing-e2e-worker-w5-4cce33-56666d7678-r88jh Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m23s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-4cce33-66b786457f-2ggrc Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 4m21s Normal ScalingReplicaSet deployment/e2e-worker-w5-4cce33-openshift-default Scaled up replica set e2e-worker-w5-4cce33-openshift-default-747768654f from 4 to 8 4m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 4m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d 4m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g 4m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 4m20s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Add eth0 [10.134.0.65/23] from ovn-kubernetes 4m20s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m20s Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd 4m20s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Container created 4m20s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4r776 Stopping container istio-proxy 4m20s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Stopping container istio-proxy 4m20s Normal SuccessfulCreate replicaset/e2e-worker-w6-cedfca-openshift-default-6b99d7696f Created pod: e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 4m20s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-mfsjl Stopping container istio-proxy 4m20s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Container started 4m20s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Container created 4m20s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Container started 4m20s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Container started 4m20s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m20s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-cedfca-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 4m20s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Container created 4m20s Normal ScalingReplicaSet deployment/e2e-worker-w6-cedfca-openshift-default Scaled up replica set e2e-worker-w6-cedfca-openshift-default-6b99d7696f from 4 to 6 4m20s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Add eth0 [10.134.0.66/23] from ovn-kubernetes 4m20s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-m6qb5 Stopping container istio-proxy 4m20s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m20s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Add eth0 [10.133.0.86/23] from ovn-kubernetes 4m20s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m20s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Add eth0 [10.133.0.85/23] from ovn-kubernetes 4m19s Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Container created 4m19s Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m19s Normal Pulled pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m19s Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Container started 4m19s Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Add eth0 [10.134.0.67/23] from ovn-kubernetes 4m19s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Stopping container istio-proxy 4m19s Normal Created pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-4qlhd Container created 4m19s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Container created 4m19s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Container started 4m19s Normal Killing pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Stopping container istio-proxy 4m19s Normal AddedInterface pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Add eth0 [10.133.0.87/23] from ovn-kubernetes 4m19s Normal Started pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lptr9 Container started 4m15s Warning ProbeError pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Readiness probe error: HTTP probe failed with statuscode: 503... 4m15s Warning Unhealthy pod/e2e-worker-w6-cedfca-openshift-default-6b99d7696f-lg5qh Readiness probe failed: HTTP probe failed with statuscode: 503 4m13s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Startup probe failed: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused 4m13s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Startup probe error: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused... 4m13s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Startup probe failed: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused 4m13s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Startup probe error: Get "http://10.133.0.86:15021/healthz/ready": dial tcp 10.133.0.86:15021: connect: connection refused... 4m13s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Startup probe error: Get "http://10.133.0.85:15021/healthz/ready": dial tcp 10.133.0.85:15021: connect: connection refused... 4m13s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Startup probe error: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused... 4m13s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Startup probe failed: Get "http://10.134.0.65:15021/healthz/ready": dial tcp 10.134.0.65:15021: connect: connection refused 4m13s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Startup probe failed: Get "http://10.134.0.66:15021/healthz/ready": dial tcp 10.134.0.66:15021: connect: connection refused 4m5s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Container started 4m5s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Container created 4m5s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m5s Normal Created pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Container created 4m5s Normal ScalingReplicaSet deployment/e2e-worker-w5-4cce33-openshift-default Scaled up replica set e2e-worker-w5-4cce33-openshift-default-747768654f from 8 to 10 4m5s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-4cce33-openshift-default New size: 10; reason: 4m5s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f (combined from similar events): Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 4m5s Normal SuccessfulCreate replicaset/e2e-worker-w5-4cce33-openshift-default-747768654f Created pod: e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn 4m5s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Add eth0 [10.132.0.80/23] from ovn-kubernetes 4m5s Normal Started pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Container started 4m5s Normal AddedInterface pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Add eth0 [10.134.0.68/23] from ovn-kubernetes 4m5s Normal Pulled pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-8g4r6 Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-6czgn Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-znf89 Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Stopping container istio-proxy 4m Normal Killing pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Stopping container istio-proxy 3m59s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Readiness probe error: HTTP probe failed with statuscode: 503... 3m59s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-njdgq Readiness probe failed: HTTP probe failed with statuscode: 503 3m58s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Readiness probe error: HTTP probe failed with statuscode: 503... 3m58s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-mpnbg Readiness probe failed: HTTP probe failed with statuscode: 503 3m58s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Readiness probe error: HTTP probe failed with statuscode: 503... 3m58s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-rhsz6 Readiness probe failed: HTTP probe failed with statuscode: 503 3m57s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Readiness probe failed: HTTP probe failed with statuscode: 503 3m57s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Readiness probe failed: HTTP probe failed with statuscode: 503 3m57s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-85pc8 Readiness probe error: HTTP probe failed with statuscode: 503... 3m57s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Readiness probe error: HTTP probe failed with statuscode: 503... 3m57s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-x6hf9 Readiness probe failed: HTTP probe failed with statuscode: 503 3m57s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-9vd9d Readiness probe error: HTTP probe failed with statuscode: 503... 3m56s Warning ProbeError pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Readiness probe error: HTTP probe failed with statuscode: 503... 3m56s Warning Unhealthy pod/e2e-worker-w5-4cce33-openshift-default-747768654f-tkn4g Readiness probe failed: HTTP probe failed with statuscode: 503 3m23s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 2; reason: All metrics below target 3m23s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-6886b96f9 from 3 to 2 3m23s Normal Killing pod/maas-default-gateway-openshift-default-6886b96f9-bnxlc Stopping container istio-proxy 3m23s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-6886b96f9 Deleted pod: maas-default-gateway-openshift-default-6886b96f9-bnxlc