=== events in openshift-ingress (2026-11-04T05:49:15Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 10m Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf to ip-10-0-139-34.us-west-2.compute.internal 3m20s Normal Scheduled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Successfully assigned openshift-ingress/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 to ip-10-0-133-143.us-west-2.compute.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5s5sq to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 to ip-10-0-133-143.us-west-2.compute.internal 3m35s Normal Scheduled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Successfully assigned openshift-ingress/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 to ip-10-0-130-185.us-west-2.compute.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5gwkz to ip-10-0-139-34.us-west-2.compute.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-hrddl to ip-10-0-139-34.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm to ip-10-0-130-185.us-west-2.compute.internal 6m21s Normal Scheduled pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Successfully assigned openshift-ingress/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb to ip-10-0-130-185.us-west-2.compute.internal 2m48s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw to ip-10-0-130-185.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m to ip-10-0-130-185.us-west-2.compute.internal 12m Normal Scheduled pod/payload-pre-processing-7b98bb678c-chjgw Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-chjgw to ip-10-0-133-143.us-west-2.compute.internal 3m35s Normal Scheduled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Successfully assigned openshift-ingress/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq to ip-10-0-139-34.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k to ip-10-0-130-185.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Successfully assigned openshift-ingress/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx to ip-10-0-139-34.us-west-2.compute.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-jrfvz to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn to ip-10-0-130-185.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 to ip-10-0-130-185.us-west-2.compute.internal 7m38s Normal Scheduled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Successfully assigned openshift-ingress/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f to ip-10-0-133-143.us-west-2.compute.internal 2m33s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp to ip-10-0-130-185.us-west-2.compute.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-s2524 to ip-10-0-133-143.us-west-2.compute.internal 3m18s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm to ip-10-0-133-143.us-west-2.compute.internal 16m Normal Scheduled pod/kube-auth-proxy-5c6f4696b9-vlb8m Successfully assigned openshift-ingress/kube-auth-proxy-5c6f4696b9-vlb8m to ip-10-0-130-185.us-west-2.compute.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-t77lc to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx to ip-10-0-130-185.us-west-2.compute.internal 18m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf to ip-10-0-139-34.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Successfully assigned openshift-ingress/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm to ip-10-0-130-185.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Successfully assigned openshift-ingress/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm to ip-10-0-133-143.us-west-2.compute.internal 3m18s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c to ip-10-0-130-185.us-west-2.compute.internal 2m48s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Successfully assigned openshift-ingress/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd to ip-10-0-130-185.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf to ip-10-0-130-185.us-west-2.compute.internal 4m21s Normal Scheduled pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Successfully assigned openshift-ingress/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Successfully assigned openshift-ingress/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 to ip-10-0-130-185.us-west-2.compute.internal 4m4s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 to ip-10-0-130-185.us-west-2.compute.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-xbdch to ip-10-0-130-185.us-west-2.compute.internal 16m Normal Scheduled pod/kube-auth-proxy-5c6f4696b9-jdr4t Successfully assigned openshift-ingress/kube-auth-proxy-5c6f4696b9-jdr4t to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f to ip-10-0-133-143.us-west-2.compute.internal 18m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-9cbtq Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-9cbtq to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Successfully assigned openshift-ingress/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr to ip-10-0-139-34.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt to ip-10-0-130-185.us-west-2.compute.internal 12m Normal Scheduled pod/payload-processing-7bb9489676-4rp6q Successfully assigned openshift-ingress/payload-processing-7bb9489676-4rp6q to ip-10-0-130-185.us-west-2.compute.internal 4m34s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 to ip-10-0-139-34.us-west-2.compute.internal 18m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 to ip-10-0-139-34.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 to ip-10-0-130-185.us-west-2.compute.internal 29m Warning FailedScheduling pod/router-default-58c967f969-4xztc no nodes available to schedule pods 2m3s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 to ip-10-0-133-143.us-west-2.compute.internal 22m Warning FailedScheduling pod/router-default-58c967f969-4xztc no nodes available to schedule pods 22m Normal Scheduled pod/router-default-58c967f969-4xztc Successfully assigned openshift-ingress/router-default-58c967f969-4xztc to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk to ip-10-0-139-34.us-west-2.compute.internal 10m Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj to ip-10-0-133-143.us-west-2.compute.internal 15m Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-z4xcr Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z4xcr to ip-10-0-133-143.us-west-2.compute.internal 2m3s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 to ip-10-0-133-143.us-west-2.compute.internal 10m Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw to ip-10-0-130-185.us-west-2.compute.internal 16m Normal Scheduled pod/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh to ip-10-0-139-34.us-west-2.compute.internal 6m5s Normal Scheduled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Successfully assigned openshift-ingress/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s to ip-10-0-133-143.us-west-2.compute.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-vmk4n to ip-10-0-130-185.us-west-2.compute.internal 12m Normal Scheduled pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8spv4 to ip-10-0-130-185.us-west-2.compute.internal 29m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-58c967f969 from 0 to 1 29m Normal SuccessfulCreate replicaset/router-default-58c967f969 Created pod: router-default-58c967f969-4xztc 28m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 28m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 28m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 22m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 22m Warning FailedMount pod/router-default-58c967f969-4xztc MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 22m Warning FailedMount pod/router-default-58c967f969-4xztc MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 22m Warning FailedMount pod/router-default-58c967f969-4xztc MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 22m Warning FailedMount pod/router-default-58c967f969-4xztc MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 21m Normal Created pod/router-default-58c967f969-4xztc Container created 21m Normal Started pod/router-default-58c967f969-4xztc Container started 21m Normal Pulled pod/router-default-58c967f969-4xztc Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 21m Normal AddedInterface pod/router-default-58c967f969-4xztc Add eth0 [10.132.0.17/23] from ovn-kubernetes 18m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 18m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 18m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-9cbtq 18m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-9cbtq Add eth0 [10.132.0.26/23] from ovn-kubernetes 18m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-9cbtq Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 18m Normal Started pod/istiod-openshift-gateway-7876bfcb74-9cbtq Container started 18m Normal Created pod/istiod-openshift-gateway-7876bfcb74-9cbtq Container created 18m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-9cbtq Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 5.131s (5.131s including waiting). Image size: 185476477 bytes. 18m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-75b555675f from 0 to 1 18m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-778b7b5c47 from 0 to 1 18m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 18m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-9cbtq Readiness probe failed: HTTP probe failed with statuscode: 503 18m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-75b555675f Created pod: openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf 18m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-9cbtq Readiness probe error: HTTP probe failed with statuscode: 503... 18m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-tmnh8 17m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 17m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Add eth0 [10.133.0.26/23] from ovn-kubernetes 17m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Add eth0 [10.133.0.25/23] from ovn-kubernetes 17m Normal Pulling pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 17m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 17m Normal Created pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Container created 17m Normal Started pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Container started 17m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-75b555675f-hh8sf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 10.953s (10.953s including waiting). Image size: 183555333 bytes. 17m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 11.304s (11.304s including waiting). Image size: 183555333 bytes. 17m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 17m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Normal Pulling pod/kube-auth-proxy-5c6f4696b9-jdr4t Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 16m Normal SuccessfulCreate replicaset/kube-auth-proxy-5c6f4696b9 Created pod: kube-auth-proxy-5c6f4696b9-jdr4t 16m Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-5c6f4696b9 from 0 to 2 16m Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-58c98c4485 from 0 to 1 16m Normal SuccessfulCreate replicaset/kube-auth-proxy-5c6f4696b9 Created pod: kube-auth-proxy-5c6f4696b9-vlb8m 16m Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-58c98c4485 Created pod: data-science-gateway-data-science-gateway-class-58c98c4485xfwsh 16m Normal AddedInterface pod/kube-auth-proxy-5c6f4696b9-vlb8m Add eth0 [10.134.0.26/23] from ovn-kubernetes 16m Normal Pulling pod/kube-auth-proxy-5c6f4696b9-vlb8m Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 16m Normal AddedInterface pod/kube-auth-proxy-5c6f4696b9-jdr4t Add eth0 [10.132.0.28/23] from ovn-kubernetes 16m Normal Pulled pod/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 16m Normal Created pod/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh Container created 16m Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh Add eth0 [10.133.0.34/23] from ovn-kubernetes 16m Normal Started pod/data-science-gateway-data-science-gateway-class-58c98c4485xfwsh Container started 16m Normal Pulled pod/kube-auth-proxy-5c6f4696b9-jdr4t Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 3.795s (3.795s including waiting). Image size: 188904677 bytes. 16m Normal Started pod/kube-auth-proxy-5c6f4696b9-jdr4t Container started 16m Normal Created pod/kube-auth-proxy-5c6f4696b9-jdr4t Container created 16m Normal Created pod/kube-auth-proxy-5c6f4696b9-vlb8m Container created 16m Normal Started pod/kube-auth-proxy-5c6f4696b9-vlb8m Container started 16m Normal Pulled pod/kube-auth-proxy-5c6f4696b9-vlb8m Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 5.501s (5.501s including waiting). Image size: 188904677 bytes. 16m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 16m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 16m Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 15m Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 15m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 15m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 15m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-z4xcr 15m Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-z4xcr Add eth0 [10.132.0.31/23] from ovn-kubernetes 15m Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-z4xcr Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 15m Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-z4xcr Container created 15m Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-z4xcr Container started 15m Normal Killing pod/istiod-openshift-gateway-7876bfcb74-9cbtq Stopping container discovery 15m Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-9cbtq 15m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 14m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 14m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Container started 14m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Container created 14m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Startup probe error: Get "http://10.133.0.25:15021/healthz/ready": dial tcp 10.133.0.25:15021: connect: connection refused... 14m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-tmnh8 Startup probe failed: Get "http://10.133.0.25:15021/healthz/ready": dial tcp 10.133.0.25:15021: connect: connection refused 14m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 14m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 14m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-5s5sq 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-778b7b5c47 from 1 to 3 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-vmk4n 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Add eth0 [10.132.0.41/23] from ovn-kubernetes 12m Normal Pulling pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Add eth0 [10.134.0.33/23] from ovn-kubernetes 12m Normal Pulling pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 12m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Container created 12m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Container created 12m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Container started 12m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.726s (4.726s including waiting). Image size: 183555333 bytes. 12m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 4.467s (4.467s including waiting). Image size: 183555333 bytes. 12m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Container started 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Startup probe failed: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Startup probe failed: Get "http://10.134.0.33:15021/healthz/ready": dial tcp 10.134.0.33:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Startup probe error: Get "http://10.132.0.41:15021/healthz/ready": dial tcp 10.132.0.41:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Startup probe error: Get "http://10.134.0.33:15021/healthz/ready": dial tcp 10.134.0.33:15021: connect: connection refused... 12m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 12m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-778b7b5c47 from 3 to 6 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-8spv4 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-hrddl 12m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-s2524 12m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Container created 12m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Container created 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Add eth0 [10.134.0.35/23] from ovn-kubernetes 12m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Container started 12m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Container created 12m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Add eth0 [10.133.0.47/23] from ovn-kubernetes 12m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Add eth0 [10.132.0.42/23] from ovn-kubernetes 12m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 12m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Container started 12m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Container started 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Startup probe failed: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Startup probe failed: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-8spv4 Startup probe error: Get "http://10.134.0.35:15021/healthz/ready": dial tcp 10.134.0.35:15021: connect: connection refused... 12m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Startup probe error: Get "http://10.132.0.42:15021/healthz/ready": dial tcp 10.132.0.42:15021: connect: connection refused... 12m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Startup probe failed: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused 12m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-hrddl Startup probe error: Get "http://10.133.0.47:15021/healthz/ready": dial tcp 10.133.0.47:15021: connect: connection refused... 12m Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-7bb9489676 from 0 to 1 12m Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 12m Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 12m Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-chjgw 12m Normal SuccessfulCreate replicaset/payload-processing-7bb9489676 Created pod: payload-processing-7bb9489676-4rp6q 12m Warning FailedCreate replicaset/payload-processing-7bb9489676 Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 11m Normal Pulling pod/payload-pre-processing-7b98bb678c-chjgw Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 11m Normal AddedInterface pod/payload-processing-7bb9489676-4rp6q Add eth0 [10.134.0.36/23] from ovn-kubernetes 11m Normal Pulling pod/payload-processing-7bb9489676-4rp6q Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 11m Normal AddedInterface pod/payload-pre-processing-7b98bb678c-chjgw Add eth0 [10.132.0.44/23] from ovn-kubernetes 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Readiness probe error: Get "http://10.134.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Readiness probe error: Get "http://10.132.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Readiness probe failed: Get "http://10.134.0.33:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Readiness probe failed: Get "http://10.132.0.41:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 11m Normal Pulled pod/payload-pre-processing-7b98bb678c-chjgw Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.369s (3.369s including waiting). Image size: 125353963 bytes. 11m Normal Created pod/payload-pre-processing-7b98bb678c-chjgw Container created 11m Normal Pulled pod/payload-processing-7bb9489676-4rp6q Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.13s (3.13s including waiting). Image size: 125353963 bytes. 11m Normal Started pod/payload-pre-processing-7b98bb678c-chjgw Container started 11m Normal Created pod/payload-processing-7bb9489676-4rp6q Container created 11m Normal Started pod/payload-processing-7bb9489676-4rp6q Container started 11m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-778b7b5c47 from 6 to 10 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-5gwkz 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-t77lc 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 Created pod: maas-default-gateway-openshift-default-778b7b5c47-xbdch 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-778b7b5c47 (combined from similar events): Created pod: maas-default-gateway-openshift-default-778b7b5c47-jrfvz 11m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Add eth0 [10.132.0.46/23] from ovn-kubernetes 11m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Add eth0 [10.132.0.45/23] from ovn-kubernetes 11m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Add eth0 [10.134.0.38/23] from ovn-kubernetes 11m Normal AddedInterface pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Add eth0 [10.133.0.49/23] from ovn-kubernetes 11m Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 11m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Container started 11m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Container created 11m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Container started 11m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Container created 11m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Container started 11m Normal Started pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Container started 11m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Container created 11m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Pulled pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 11m Normal Created pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Container created 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Startup probe error: Get "http://10.132.0.46:15021/healthz/ready": dial tcp 10.132.0.46:15021: connect: connection refused... 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": dial tcp 10.132.0.46:15021: connect: connection refused 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused 11m Warning Unhealthy pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-t77lc Startup probe error: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused... 11m Warning ProbeError pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Startup probe error: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused... 10m Normal AddedLabel gateway/e2e-worker-w5-283815 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-283815 10m Normal AddedLabel gateway/e2e-worker-w6-fbbad9 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-fbbad9 10m Normal AddedLabel gateway/e2e-worker-w2-961485 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-961485 10m Normal AddedLabel gateway/e2e-worker-w3-a84d48 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-a84d48 10m Normal AddedLabel gateway/e2e-worker-w1-098178 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-098178 10m Normal AddedLabel gateway/e2e-worker-w0-90dc8d Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-90dc8d 10m Normal AddedLabel gateway/e2e-worker-w4-901a00 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-901a00 10m Normal SuccessfulCreate replicaset/e2e-worker-w3-a84d48-openshift-default-d8668c897 Created pod: e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm 10m Normal ScalingReplicaSet deployment/e2e-worker-w0-90dc8d-openshift-default Scaled up replica set e2e-worker-w0-90dc8d-openshift-default-6fb649d46f from 0 to 1 10m Normal SuccessfulCreate replicaset/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65 Created pod: e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 10m Normal AddedInterface pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Add eth0 [10.134.0.40/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/e2e-worker-w1-098178-openshift-default Scaled up replica set e2e-worker-w1-098178-openshift-default-6b946b56b5 from 0 to 1 10m Normal SuccessfulCreate replicaset/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f Created pod: e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm 10m Normal AddedInterface pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Add eth0 [10.134.0.39/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/e2e-worker-w3-a84d48-openshift-default Scaled up replica set e2e-worker-w3-a84d48-openshift-default-d8668c897 from 0 to 1 10m Normal ScalingReplicaSet deployment/e2e-worker-w6-fbbad9-openshift-default Scaled up replica set e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65 from 0 to 1 10m Normal ScalingReplicaSet deployment/e2e-worker-w4-901a00-openshift-default Scaled up replica set e2e-worker-w4-901a00-openshift-default-57f697764c from 0 to 1 10m Normal AddedInterface pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Add eth0 [10.132.0.47/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf 10m Normal ScalingReplicaSet deployment/e2e-worker-w5-283815-openshift-default Scaled up replica set e2e-worker-w5-283815-openshift-default-578d454689 from 0 to 1 10m Normal AddedInterface pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Add eth0 [10.132.0.48/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/e2e-worker-w1-098178-openshift-default-6b946b56b5 Created pod: e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm 10m Normal SuccessfulCreate replicaset/e2e-worker-w5-283815-openshift-default-578d454689 Created pod: e2e-worker-w5-283815-openshift-default-578d454689-qw4rd 10m Normal AddedInterface pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Add eth0 [10.134.0.41/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw 10m Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Add eth0 [10.133.0.50/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 0 to 1 10m Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Add eth0 [10.134.0.42/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-fbbad9 Scaled up replica set payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-fbbad9 Scaled up replica set payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc Created pod: payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd Created pod: payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd Error creating: pods "payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-fbbad9: serviceaccount "payload-processing-e2e-worker-w6-fbbad9" not found 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Add eth0 [10.132.0.49/23] from ovn-kubernetes 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc Error creating: pods "payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-fbbad9: serviceaccount "payload-processing-e2e-worker-w6-fbbad9" not found 10m Normal Created pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Container created 10m Normal Started pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Container started 10m Normal Pulled pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Container started 10m Normal AddedInterface pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Add eth0 [10.134.0.43/23] from ovn-kubernetes 10m Normal Created pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Container created 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-a84d48-55855658b Error creating: pods "payload-processing-e2e-worker-w3-a84d48-55855658b-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-a84d48: serviceaccount "payload-processing-e2e-worker-w3-a84d48" not found 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879 Created pod: payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-a84d48 Scaled up replica set payload-processing-e2e-worker-w3-a84d48-55855658b from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-a84d48-55855658b Created pod: payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-a84d48 Scaled up replica set payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879 from 0 to 1 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879 Error creating: pods "payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-a84d48: serviceaccount "payload-processing-e2e-worker-w3-a84d48" not found 10m Normal Started pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Container started 10m Normal Created pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Container created 10m Normal Started pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Container started 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Pulled pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Add eth0 [10.132.0.51/23] from ovn-kubernetes 10m Normal Created pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Container created 10m Normal AddedInterface pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Add eth0 [10.132.0.52/23] from ovn-kubernetes 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-098178 Scaled up replica set payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-098178 Scaled up replica set payload-processing-e2e-worker-w1-098178-5bb799977d from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-098178-5bb799977d Created pod: payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c Created pod: payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c Error creating: pods "payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-098178: serviceaccount "payload-processing-e2e-worker-w1-098178" not found 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-098178-5bb799977d Error creating: pods "payload-processing-e2e-worker-w1-098178-5bb799977d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-098178: serviceaccount "payload-processing-e2e-worker-w1-098178" not found 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Add eth0 [10.134.0.44/23] from ovn-kubernetes 10m Normal Started pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Container started 10m Normal Started pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Container started 10m Normal Created pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Container created 10m Normal Pulled pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal AddedInterface pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Add eth0 [10.134.0.45/23] from ovn-kubernetes 10m Normal Created pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Container created 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-90dc8d Scaled up replica set payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f from 0 to 1 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-90dc8d-665d558cfd Error creating: pods "payload-processing-e2e-worker-w0-90dc8d-665d558cfd-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-90dc8d: serviceaccount "payload-processing-e2e-worker-w0-90dc8d" not found 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-90dc8d-665d558cfd Created pod: payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-90dc8d Scaled up replica set payload-processing-e2e-worker-w0-90dc8d-665d558cfd from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f Created pod: payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f Error creating: pods "payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-90dc8d: serviceaccount "payload-processing-e2e-worker-w0-90dc8d" not found 10m Normal Created pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Container created 10m Normal Created pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Container created 10m Normal AddedInterface pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Add eth0 [10.134.0.47/23] from ovn-kubernetes 10m Normal Pulled pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Container started 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Add eth0 [10.134.0.46/23] from ovn-kubernetes 10m Normal Started pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Container started 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67 Error creating: pods "payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-901a00: serviceaccount "payload-processing-e2e-worker-w4-901a00" not found 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6 Error creating: pods "payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-901a00: serviceaccount "payload-processing-e2e-worker-w4-901a00" not found 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-901a00 Scaled up replica set payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-901a00 Scaled up replica set payload-processing-e2e-worker-w4-901a00-86cc8cfdc6 from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67 Created pod: payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf 10m Normal Pulled pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Container started 10m Normal Created pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Container created 10m Normal AddedInterface pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Add eth0 [10.134.0.49/23] from ovn-kubernetes 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6 Created pod: payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Add eth0 [10.134.0.48/23] from ovn-kubernetes 10m Normal Pulled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 10m Normal Started pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Container started 10m Normal Started pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Container started 10m Normal Created pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Container created 10m Normal Created pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Container created 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc Created pod: payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-961485-78dd6864c6 Error creating: pods "payload-processing-e2e-worker-w2-961485-78dd6864c6-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-961485: serviceaccount "payload-processing-e2e-worker-w2-961485" not found 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc Error creating: pods "payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-961485: serviceaccount "payload-processing-e2e-worker-w2-961485" not found 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-961485 Scaled up replica set payload-processing-e2e-worker-w2-961485-78dd6864c6 from 0 to 1 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-961485 Scaled up replica set payload-pre-processing-e2e-worker-w2-961485-5d688b9bc from 0 to 1 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-961485-78dd6864c6 Created pod: payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Created pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Container created 10m Normal Started pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Container started 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Add eth0 [10.132.0.55/23] from ovn-kubernetes 10m Normal Created pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Container created 10m Normal AddedInterface pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Add eth0 [10.132.0.56/23] from ovn-kubernetes 10m Normal Started pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Container started 10m Normal Pulled pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-283815 Scaled up replica set payload-processing-e2e-worker-w5-283815-79598958bb from 0 to 1 10m Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59 Error creating: pods "payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-283815: serviceaccount "payload-processing-e2e-worker-w5-283815" not found 10m Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59 Created pod: payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f 10m Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-283815 Scaled up replica set payload-pre-processing-e2e-worker-w5-283815-77ccd5c59 from 0 to 1 10m Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-283815-79598958bb Error creating: pods "payload-processing-e2e-worker-w5-283815-79598958bb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-283815: serviceaccount "payload-processing-e2e-worker-w5-283815" not found 10m Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-283815-79598958bb Created pod: payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 10m Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Add eth0 [10.132.0.57/23] from ovn-kubernetes 10m Normal Started pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Container started 10m Normal Pulled pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Created pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Container created 10m Normal Started pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Container started 10m Normal AddedInterface pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Add eth0 [10.134.0.50/23] from ovn-kubernetes 10m Normal Pulled pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 10m Normal Created pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Container created 10m Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 10m Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 9m59s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Container started 9m59s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Container created 9m59s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Container created 9m59s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Container started 9m59s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m59s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m54s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Startup probe error: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused... 9m54s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Startup probe failed: Get "http://10.134.0.42:15021/healthz/ready": dial tcp 10.134.0.42:15021: connect: connection refused 9m54s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m54s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m53s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Startup probe error: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused... 9m53s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Startup probe failed: Get "http://10.133.0.50:15021/healthz/ready": dial tcp 10.133.0.50:15021: connect: connection refused 9m52s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m52s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m50s Warning Unhealthy pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Readiness probe failed: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 9m50s Warning ProbeError pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Readiness probe error: Get "http://10.132.0.48:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 9m49s Normal Created pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Container created 9m49s Normal Started pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Container started 9m49s Normal Pulled pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m43s Warning Unhealthy pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused 9m43s Warning ProbeError pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Startup probe error: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused... 9m39s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m39s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m38s Normal Started pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Container started 9m38s Normal Started pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Container started 9m38s Normal Pulled pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m38s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-098178-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m38s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-098178-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m38s Normal Pulled pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m38s Normal Created pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Container created 9m38s Normal Created pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Container created 9m37s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m37s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m33s Warning ProbeError pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Startup probe error: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused... 9m33s Warning Unhealthy pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Startup probe failed: Get "http://10.134.0.41:15021/healthz/ready": dial tcp 10.134.0.41:15021: connect: connection refused 9m32s Warning Unhealthy pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Startup probe failed: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused 9m32s Warning ProbeError pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Startup probe error: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused... 9m29s Normal Created pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Container created 9m29s Normal Pulled pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m29s Normal Started pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Container started 9m23s Warning Unhealthy pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused 9m23s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-fbbad9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m23s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-fbbad9-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m23s Warning ProbeError pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused... 9m22s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m22s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m22s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-283815-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m22s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-283815-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m22s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m22s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m7s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-283815-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m7s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-283815-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 8m53s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-90dc8d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m53s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-fbbad9-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m53s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-90dc8d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m53s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-098178-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m53s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-098178-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m53s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-fbbad9-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m38s Normal ScalingReplicaSet deployment/e2e-worker-w3-a84d48-openshift-default Scaled up replica set e2e-worker-w3-a84d48-openshift-default-d8668c897 from 1 to 2 7m38s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 7m38s Normal SuccessfulCreate replicaset/e2e-worker-w3-a84d48-openshift-default-d8668c897 Created pod: e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv 7m37s Normal AddedInterface pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Add eth0 [10.132.0.60/23] from ovn-kubernetes 7m37s Normal Started pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Container started 7m37s Normal Created pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Container created 7m37s Normal Pulled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 7m30s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Startup probe error: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused... 7m30s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Startup probe failed: Get "http://10.132.0.60:15021/healthz/ready": dial tcp 10.132.0.60:15021: connect: connection refused 6m24s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-s2524 Stopping container istio-proxy 6m24s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-jrfvz Stopping container istio-proxy 6m24s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-778b7b5c47 from 10 to 7 6m24s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-5s5sq Stopping container istio-proxy 6m24s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 7; reason: All metrics below target 6m24s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-5s5sq 6m24s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-jrfvz 6m24s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-s2524 6m22s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-90dc8d-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 6m22s Normal ScalingReplicaSet deployment/e2e-worker-w0-90dc8d-openshift-default Scaled up replica set e2e-worker-w0-90dc8d-openshift-default-6fb649d46f from 1 to 2 6m21s Normal SuccessfulCreate replicaset/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f Created pod: e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb 6m21s Normal AddedInterface pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Add eth0 [10.134.0.56/23] from ovn-kubernetes 6m21s Normal Pulled pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m21s Normal Started pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Container started 6m21s Normal Created pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Container created 6m15s Warning ProbeError pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Startup probe error: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused... 6m15s Warning Unhealthy pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Startup probe failed: Get "http://10.134.0.56:15021/healthz/ready": dial tcp 10.134.0.56:15021: connect: connection refused 6m5s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 6m5s Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 1 to 2 6m5s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s 6m4s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Add eth0 [10.132.0.62/23] from ovn-kubernetes 6m4s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m4s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Container created 6m4s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Container started 5m58s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Startup probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused 5m58s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Startup probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused... 4m44s Normal Killing pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Stopping container payload-pre-processing 4m44s Normal Killing pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Stopping container payload-processing 4m39s Normal Killing pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Stopping container payload-pre-processing 4m39s Normal Killing pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Stopping container payload-processing 4m39s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 5; reason: All metrics below target 4m39s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-vmk4n 4m39s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-778b7b5c47 from 7 to 5 4m39s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-xbdch 4m38s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-xbdch Stopping container istio-proxy 4m38s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-vmk4n Stopping container istio-proxy 4m34s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 4m34s Normal Killing pod/payload-processing-e2e-worker-w0-90dc8d-665d558cfd-gvxfn Stopping container payload-processing 4m34s Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 2 to 3 4m34s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Add eth0 [10.133.0.64/23] from ovn-kubernetes 4m34s Normal Killing pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Stopping container payload-pre-processing 4m34s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 4m33s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m33s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Container created 4m33s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Container started 4m32s Normal Killing pod/payload-processing-e2e-worker-w6-fbbad9-d6f88b5fd-wsdqx Stopping container payload-processing 4m31s Normal Killing pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Stopping container payload-pre-processing 4m27s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 4m27s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 4m23s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: All metrics below target 4m23s Normal SuccessfulDelete replicaset/maas-default-gateway-openshift-default-778b7b5c47 Deleted pod: maas-default-gateway-openshift-default-778b7b5c47-5gwkz 4m23s Normal Killing pod/maas-default-gateway-openshift-default-778b7b5c47-5gwkz Stopping container istio-proxy 4m23s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-283815-77ccd5c59-dpv4f Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m23s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled down replica set maas-default-gateway-openshift-default-778b7b5c47 from 5 to 4 4m22s Warning Unhealthy pod/payload-processing-e2e-worker-w5-283815-79598958bb-xdnd2 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m21s Normal SuccessfulCreate replicaset/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65 Created pod: e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px 4m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-fbbad9-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 4m21s Normal ScalingReplicaSet deployment/e2e-worker-w6-fbbad9-openshift-default Scaled up replica set e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65 from 1 to 2 4m20s Normal Created pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Container created 4m20s Normal Started pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Container started 4m20s Normal Pulled pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m20s Normal AddedInterface pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Add eth0 [10.132.0.67/23] from ovn-kubernetes 4m19s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-098178-6b4f56d54c-nk29m Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m19s Warning Unhealthy pod/payload-processing-e2e-worker-w1-098178-5bb799977d-r46pm Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m16s Normal Killing pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Stopping container istio-proxy 4m14s Normal Killing pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-6rckb Stopping container istio-proxy 4m14s Normal Killing pod/e2e-worker-w0-90dc8d-openshift-default-6fb649d46f-l2zsm Stopping container istio-proxy 4m13s Warning Unhealthy pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused 4m13s Warning ProbeError pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Startup probe error: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused... 4m13s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-90dc8d-7db59b567f-5dkzt Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m12s Normal Killing pod/e2e-worker-w5-283815-openshift-default-578d454689-qw4rd Stopping container istio-proxy 4m12s Warning Unhealthy pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Readiness probe failed: HTTP probe failed with statuscode: 503 4m12s Warning ProbeError pod/e2e-worker-w1-098178-openshift-default-6b946b56b5-6dxhm Readiness probe error: HTTP probe failed with statuscode: 503... 4m10s Normal Killing pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Stopping container istio-proxy 4m10s Normal Killing pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-kx8px Stopping container istio-proxy 4m7s Warning ProbeError pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Readiness probe error: HTTP probe failed with statuscode: 503... 4m7s Warning Unhealthy pod/e2e-worker-w6-fbbad9-openshift-default-59cf5b7f65-rwph4 Readiness probe failed: HTTP probe failed with statuscode: 503 4m7s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-fbbad9-5cbd955cbc-7bdcj Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 4m4s Normal ScalingReplicaSet deployment/e2e-worker-w4-901a00-openshift-default Scaled up replica set e2e-worker-w4-901a00-openshift-default-57f697764c from 1 to 2 4m4s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 4m4s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 4m3s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Container created 4m3s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Container started 4m3s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Add eth0 [10.134.0.67/23] from ovn-kubernetes 4m3s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m57s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Startup probe failed: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused 3m57s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Startup probe error: Get "http://10.134.0.67:15021/healthz/ready": dial tcp 10.134.0.67:15021: connect: connection refused... 3m37s Normal Killing pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Stopping container payload-processing 3m36s Normal Killing pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Stopping container payload-pre-processing 3m36s Normal SuccessfulCreate replicaset/e2e-worker-w3-a84d48-openshift-default-d8668c897 Created pod: e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq 3m36s Normal ScalingReplicaSet deployment/e2e-worker-w3-a84d48-openshift-default Scaled up replica set e2e-worker-w3-a84d48-openshift-default-d8668c897 from 2 to 4 3m36s Normal SuccessfulCreate replicaset/e2e-worker-w3-a84d48-openshift-default-d8668c897 Created pod: e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 3m36s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 3m35s Normal Started pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Container started 3m35s Normal AddedInterface pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Add eth0 [10.133.0.67/23] from ovn-kubernetes 3m35s Normal Pulled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m35s Normal Started pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Container started 3m35s Normal Created pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Container created 3m35s Normal Pulled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m35s Normal AddedInterface pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Add eth0 [10.134.0.71/23] from ovn-kubernetes 3m35s Normal Created pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Container created 3m28s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused... 3m28s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused... 3m28s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": dial tcp 10.133.0.67:15021: connect: connection refused 3m28s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": dial tcp 10.134.0.71:15021: connect: connection refused 3m26s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Startup probe error: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m26s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Startup probe failed: Get "http://10.133.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m26s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Startup probe failed: Get "http://10.134.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m26s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Startup probe error: Get "http://10.134.0.71:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m21s Normal ScalingReplicaSet deployment/e2e-worker-w3-a84d48-openshift-default Scaled up replica set e2e-worker-w3-a84d48-openshift-default-d8668c897 from 4 to 5 3m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-a84d48-openshift-default New size: 5; reason: 3m21s Warning Unhealthy pod/payload-processing-e2e-worker-w3-a84d48-55855658b-sl746 Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 3m21s Normal SuccessfulCreate replicaset/e2e-worker-w3-a84d48-openshift-default-d8668c897 Created pod: e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 3m21s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w3-a84d48-5bdfff8879-gp67l Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 3m20s Normal Started pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Container started 3m20s Normal AddedInterface pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Add eth0 [10.132.0.68/23] from ovn-kubernetes 3m20s Normal Pulled pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m20s Normal Created pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Container created 3m19s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm 3m19s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c 3m19s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default New size: 5; reason: cpu resource utilization (percentage of request) above target 3m19s Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 3 to 5 3m18s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m18s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Container started 3m18s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Container created 3m18s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Container started 3m18s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Add eth0 [10.134.0.72/23] from ovn-kubernetes 3m18s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Add eth0 [10.132.0.69/23] from ovn-kubernetes 3m18s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m18s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Container created 3m13s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Startup probe error: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused... 3m13s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Startup probe failed: Get "http://10.132.0.68:15021/healthz/ready": dial tcp 10.132.0.68:15021: connect: connection refused 3m11s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 3m11s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 3m11s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Startup probe error: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused... 3m11s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Startup probe failed: Get "http://10.132.0.69:15021/healthz/ready": dial tcp 10.132.0.69:15021: connect: connection refused 2m58s Normal Killing pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-mh4zv Stopping container istio-proxy 2m58s Normal Killing pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Stopping container istio-proxy 2m58s Normal Killing pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-ss6dm Stopping container istio-proxy 2m58s Normal Killing pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Stopping container istio-proxy 2m58s Normal Killing pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Stopping container istio-proxy 2m57s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Readiness probe failed: HTTP probe failed with statuscode: 503 2m57s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-x65f9 Readiness probe error: HTTP probe failed with statuscode: 503... 2m56s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Readiness probe failed: HTTP probe failed with statuscode: 503 2m56s Warning Unhealthy pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Readiness probe failed: HTTP probe failed with statuscode: 503 2m56s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-6mnq7 Readiness probe error: HTTP probe failed with statuscode: 503... 2m56s Warning ProbeError pod/e2e-worker-w3-a84d48-openshift-default-d8668c897-k5mpq Readiness probe error: HTTP probe failed with statuscode: 503... 2m49s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj 2m49s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd 2m49s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 2m49s Normal ScalingReplicaSet deployment/e2e-worker-w4-901a00-openshift-default Scaled up replica set e2e-worker-w4-901a00-openshift-default-57f697764c from 2 to 4 2m48s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m48s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Container started 2m48s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Container started 2m48s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Add eth0 [10.132.0.71/23] from ovn-kubernetes 2m48s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m48s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Add eth0 [10.132.0.70/23] from ovn-kubernetes 2m48s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Container created 2m48s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Container created 2m41s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Startup probe error: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused... 2m41s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Startup probe failed: Get "http://10.132.0.71:15021/healthz/ready": dial tcp 10.132.0.71:15021: connect: connection refused 2m41s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Startup probe failed: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused 2m41s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Startup probe error: Get "http://10.132.0.70:15021/healthz/ready": dial tcp 10.132.0.70:15021: connect: connection refused... 2m34s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp 2m34s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default New size: 5; reason: All metrics below target 2m34s Normal ScalingReplicaSet deployment/e2e-worker-w4-901a00-openshift-default Scaled up replica set e2e-worker-w4-901a00-openshift-default-57f697764c from 4 to 5 2m33s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Container created 2m33s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m33s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Container started 2m33s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Add eth0 [10.134.0.76/23] from ovn-kubernetes 2m26s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Startup probe failed: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused 2m26s Normal Killing pod/payload-pre-processing-e2e-worker-w2-961485-5d688b9bc-hg85f Stopping container payload-pre-processing 2m26s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Startup probe error: Get "http://10.134.0.76:15021/healthz/ready": dial tcp 10.134.0.76:15021: connect: connection refused... 2m26s Normal Killing pod/payload-processing-e2e-worker-w2-961485-78dd6864c6-mlkj8 Stopping container payload-processing 2m25s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Readiness probe failed: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m25s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Readiness probe error: Get "http://10.134.0.67:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m24s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Readiness probe error: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m24s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Startup probe error: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m24s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Startup probe failed: Get "http://10.134.0.76:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m24s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Readiness probe failed: Get "http://10.132.0.69:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m24s Normal Killing pod/payload-processing-e2e-worker-w4-901a00-86cc8cfdc6-rbw8k Stopping container payload-processing 2m24s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Readiness probe failed: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2m24s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Readiness probe error: Get "http://10.134.0.72:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 2m23s Normal Killing pod/payload-pre-processing-e2e-worker-w4-901a00-bc6cf6d67-gx5bf Stopping container payload-pre-processing 2m3s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-901a00-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 2m3s Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 5 to 10 2m3s Normal ScalingReplicaSet deployment/e2e-worker-w4-901a00-openshift-default Scaled up replica set e2e-worker-w4-901a00-openshift-default-57f697764c from 5 to 8 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w4-901a00-openshift-default-57f697764c Created pod: e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr 2m3s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Add eth0 [10.132.0.72/23] from ovn-kubernetes 2m3s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Add eth0 [10.133.0.69/23] from ovn-kubernetes 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk 2m3s Normal SuccessfulCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff (combined from similar events): Created pod: e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 2m3s Normal AddedInterface pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Add eth0 [10.134.0.78/23] from ovn-kubernetes 2m3s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-961485-openshift-default New size: 10; reason: cpu resource utilization (percentage of request) above target 2m2s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal ScalingReplicaSet deployment/e2e-worker-w2-961485-openshift-default Scaled up replica set e2e-worker-w2-961485-openshift-default-77d8ff86ff from 0 to 1 2m2s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Container created 2m2s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Add eth0 [10.132.0.73/23] from ovn-kubernetes 2m2s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Container started 2m2s Warning FailedCreate replicaset/e2e-worker-w2-961485-openshift-default-77d8ff86ff Error creating: pods "e2e-worker-w2-961485-openshift-default-77d8ff86ff-" is forbidden: error looking up service account openshift-ingress/e2e-worker-w2-961485-openshift-default: serviceaccount "e2e-worker-w2-961485-openshift-default" not found 2m2s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Container created 2m2s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Container started 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-2k484 Stopping container istio-proxy 2m2s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Add eth0 [10.132.0.74/23] from ovn-kubernetes 2m2s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Container created 2m2s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Container started 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-9hvq9 Stopping container istio-proxy 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Stopping container istio-proxy 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-rv5l2 Stopping container istio-proxy 2m2s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Container started 2m2s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Container created 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Stopping container istio-proxy 2m2s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Add eth0 [10.133.0.70/23] from ovn-kubernetes 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-vxbhm Stopping container istio-proxy 2m2s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Container created 2m2s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Container started 2m2s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Container created 2m2s Normal Pulled pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Created pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Container created 2m2s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Container started 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-gjflw Stopping container istio-proxy 2m2s Normal Started pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Container started 2m2s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-smw6c Stopping container istio-proxy 2m2s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Add eth0 [10.133.0.71/23] from ovn-kubernetes 2m2s Normal Pulled pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m2s Normal Created pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Container created 2m2s Normal Started pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Container started 2m2s Normal AddedInterface pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-n2mz5 Add eth0 [10.134.0.79/23] from ovn-kubernetes 2m1s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-mzmdk Stopping container istio-proxy 2m1s Normal Killing pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-wbptx Stopping container istio-proxy 117s Warning Unhealthy pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Readiness probe failed: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused 117s Warning ProbeError pod/e2e-worker-w2-961485-openshift-default-77d8ff86ff-fxz9s Readiness probe error: Get "http://10.132.0.62:15021/healthz/ready": dial tcp 10.132.0.62:15021: connect: connection refused... 116s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Startup probe failed: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Stopping container istio-proxy 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Stopping container istio-proxy 116s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Startup probe error: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused... 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Stopping container istio-proxy 116s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Readiness probe failed: HTTP probe failed with statuscode: 503 116s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-5nb25 Readiness probe error: HTTP probe failed with statuscode: 503... 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Stopping container istio-proxy 116s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Startup probe failed: Get "http://10.134.0.78:15021/healthz/ready": dial tcp 10.134.0.78:15021: connect: connection refused 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Stopping container istio-proxy 116s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-t4cr7 Startup probe error: Get "http://10.132.0.72:15021/healthz/ready": dial tcp 10.132.0.72:15021: connect: connection refused... 116s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Startup probe error: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused... 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Stopping container istio-proxy 116s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-4fxvr Startup probe failed: Get "http://10.133.0.69:15021/healthz/ready": dial tcp 10.133.0.69:15021: connect: connection refused 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-j84dw Stopping container istio-proxy 116s Normal Killing pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Stopping container istio-proxy 115s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Readiness probe error: HTTP probe failed with statuscode: 503... 115s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-sswmd Readiness probe failed: HTTP probe failed with statuscode: 503 115s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Readiness probe failed: HTTP probe failed with statuscode: 503 115s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-9nmqj Readiness probe error: HTTP probe failed with statuscode: 503... 113s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Readiness probe error: HTTP probe failed with statuscode: 503... 113s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-gd9zp Readiness probe failed: HTTP probe failed with statuscode: 503 112s Warning Unhealthy pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Readiness probe failed: HTTP probe failed with statuscode: 503 112s Warning ProbeError pod/e2e-worker-w4-901a00-openshift-default-57f697764c-2smlf Readiness probe error: HTTP probe failed with statuscode: 503...