=== events in openshift-ingress (2026-11-04T04:03:56Z) === LAST SEEN TYPE REASON OBJECT MESSAGE 9m44s Normal Scheduled pod/kube-auth-proxy-5bd469f65f-4x7zx Successfully assigned openshift-ingress/kube-auth-proxy-5bd469f65f-4x7zx to ip-10-0-129-253.us-east-2.compute.internal 4m5s Normal Scheduled pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Successfully assigned openshift-ingress/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg to ip-10-0-129-134.us-east-2.compute.internal 5m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn to ip-10-0-129-253.us-east-2.compute.internal 3m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf to ip-10-0-138-9.us-east-2.compute.internal 5m13s Normal Scheduled pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s to ip-10-0-129-134.us-east-2.compute.internal 6m29s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 to ip-10-0-129-253.us-east-2.compute.internal 2m6s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 to ip-10-0-129-134.us-east-2.compute.internal 4m6s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs to ip-10-0-129-253.us-east-2.compute.internal 6m13s Normal Scheduled pod/payload-processing-7bb9489676-sv8xh Successfully assigned openshift-ingress/payload-processing-7bb9489676-sv8xh to ip-10-0-138-9.us-east-2.compute.internal 5m14s Normal Scheduled pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp to ip-10-0-138-9.us-east-2.compute.internal 5m20s Normal Scheduled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Successfully assigned openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w to ip-10-0-129-134.us-east-2.compute.internal 5m21s Normal Scheduled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Successfully assigned openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx to ip-10-0-129-253.us-east-2.compute.internal 6m13s Normal Scheduled pod/payload-pre-processing-7b98bb678c-lqlbm Successfully assigned openshift-ingress/payload-pre-processing-7b98bb678c-lqlbm to ip-10-0-138-9.us-east-2.compute.internal 6m14s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 to ip-10-0-129-253.us-east-2.compute.internal 2m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb to ip-10-0-129-253.us-east-2.compute.internal 5m15s Normal Scheduled pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st to ip-10-0-138-9.us-east-2.compute.internal 4m5s Normal Scheduled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Successfully assigned openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h to ip-10-0-129-253.us-east-2.compute.internal 2m6s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj to ip-10-0-129-253.us-east-2.compute.internal 6m29s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 to ip-10-0-138-9.us-east-2.compute.internal 5m11s Normal Scheduled pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd to ip-10-0-129-134.us-east-2.compute.internal 6m29s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp to ip-10-0-129-253.us-east-2.compute.internal 6m14s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-gnswf to ip-10-0-129-134.us-east-2.compute.internal 2m4s Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ck24 to ip-10-0-138-9.us-east-2.compute.internal 2m21s Normal Scheduled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Successfully assigned openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 to ip-10-0-129-253.us-east-2.compute.internal 5m17s Normal Scheduled pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Successfully assigned openshift-ingress/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn to ip-10-0-138-9.us-east-2.compute.internal 3m35s Normal Scheduled pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Successfully assigned openshift-ingress/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w to ip-10-0-129-134.us-east-2.compute.internal 5m20s Normal Scheduled pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Successfully assigned openshift-ingress/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp to ip-10-0-129-253.us-east-2.compute.internal 5m11s Normal Scheduled pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Successfully assigned openshift-ingress/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r to ip-10-0-129-134.us-east-2.compute.internal 5m12s Normal Scheduled pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Successfully assigned openshift-ingress/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d to ip-10-0-129-253.us-east-2.compute.internal 5m20s Normal Scheduled pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Successfully assigned openshift-ingress/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv to ip-10-0-129-253.us-east-2.compute.internal 11m Normal Scheduled pod/istiod-openshift-gateway-7876bfcb74-lwn5j Successfully assigned openshift-ingress/istiod-openshift-gateway-7876bfcb74-lwn5j to ip-10-0-129-253.us-east-2.compute.internal 2m21s Normal Scheduled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Successfully assigned openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd to ip-10-0-138-9.us-east-2.compute.internal 5m12s Normal Scheduled pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s to ip-10-0-129-253.us-east-2.compute.internal 2m35s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 to ip-10-0-138-9.us-east-2.compute.internal 5m14s Normal Scheduled pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Successfully assigned openshift-ingress/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l to ip-10-0-129-134.us-east-2.compute.internal 14m Normal Scheduled pod/router-default-78ccf8f879-s4sx4 Successfully assigned openshift-ingress/router-default-78ccf8f879-s4sx4 to ip-10-0-138-9.us-east-2.compute.internal 15m Warning FailedScheduling pod/router-default-78ccf8f879-s4sx4 no nodes available to schedule pods 20m Warning FailedScheduling pod/router-default-78ccf8f879-s4sx4 no nodes available to schedule pods 2m20s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn to ip-10-0-138-9.us-east-2.compute.internal 3m21s Normal Scheduled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Successfully assigned openshift-ingress/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd to ip-10-0-129-134.us-east-2.compute.internal 11m Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-skgsr to ip-10-0-129-134.us-east-2.compute.internal 5m21s Normal Scheduled pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Successfully assigned openshift-ingress/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh to ip-10-0-129-253.us-east-2.compute.internal 3m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv to ip-10-0-129-134.us-east-2.compute.internal 5m17s Normal Scheduled pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs to ip-10-0-138-9.us-east-2.compute.internal 5m15s Normal Scheduled pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 to ip-10-0-138-9.us-east-2.compute.internal 3m5s Normal Scheduled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Successfully assigned openshift-ingress/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl to ip-10-0-138-9.us-east-2.compute.internal 2m20s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c to ip-10-0-129-134.us-east-2.compute.internal 6m14s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 to ip-10-0-138-9.us-east-2.compute.internal 5m13s Normal Scheduled pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Successfully assigned openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 to ip-10-0-138-9.us-east-2.compute.internal 3m51s Normal Scheduled pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Successfully assigned openshift-ingress/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc to ip-10-0-129-134.us-east-2.compute.internal 2m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s to ip-10-0-129-134.us-east-2.compute.internal 5m11s Normal Scheduled pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Successfully assigned openshift-ingress/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg to ip-10-0-138-9.us-east-2.compute.internal 5m12s Normal Scheduled pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Successfully assigned openshift-ingress/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 to ip-10-0-138-9.us-east-2.compute.internal 9m44s Normal Scheduled pod/data-science-gateway-data-science-gateway-class-597d745469j8tv2 Successfully assigned openshift-ingress/data-science-gateway-data-science-gateway-class-597d745469j8tv2 to ip-10-0-129-134.us-east-2.compute.internal 9m44s Normal Scheduled pod/kube-auth-proxy-5bd469f65f-x7t8v Successfully assigned openshift-ingress/kube-auth-proxy-5bd469f65f-x7t8v to ip-10-0-138-9.us-east-2.compute.internal 10m Normal Scheduled pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Successfully assigned openshift-ingress/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf to ip-10-0-138-9.us-east-2.compute.internal 2m20s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j to ip-10-0-129-253.us-east-2.compute.internal 6m14s Normal Scheduled pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Successfully assigned openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 to ip-10-0-129-134.us-east-2.compute.internal 2m21s Normal Scheduled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Successfully assigned openshift-ingress/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv to ip-10-0-138-9.us-east-2.compute.internal 5m21s Normal Scheduled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Successfully assigned openshift-ingress/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m to ip-10-0-129-134.us-east-2.compute.internal 8m45s Normal Scheduled pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Successfully assigned openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82hvc to ip-10-0-129-253.us-east-2.compute.internal 21m Normal ScalingReplicaSet deployment/router-default Scaled up replica set router-default-78ccf8f879 from 0 to 1 20m Normal SuccessfulCreate replicaset/router-default-78ccf8f879 Created pod: router-default-78ccf8f879-s4sx4 20m Normal EnsuringLoadBalancer service/router-default Ensuring load balancer 20m Warning UnAvailableLoadBalancer service/router-default There are no available nodes for LoadBalancer 20m Normal EnsuredLoadBalancer service/router-default Ensured load balancer 15m Normal UpdatedLoadBalancer service/router-default Updated load balancer with new hosts 14m Warning FailedMount pod/router-default-78ccf8f879-s4sx4 MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found 14m Warning FailedMount pod/router-default-78ccf8f879-s4sx4 MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 14m Warning FailedMount pod/router-default-78ccf8f879-s4sx4 MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt 14m Normal Pulled pod/router-default-78ccf8f879-s4sx4 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e" already present on machine and can be accessed by the pod 14m Normal Created pod/router-default-78ccf8f879-s4sx4 Container created 14m Normal Started pod/router-default-78ccf8f879-s4sx4 Container started 14m Normal AddedInterface pod/router-default-78ccf8f879-s4sx4 Add eth0 [10.132.0.9/23] from ovn-kubernetes 11m Normal AddedLabel gateway/maas-default-gateway Added label istio.io/rev=openshift-gateway to gateway maas-default-gateway 11m Normal AddedInterface pod/istiod-openshift-gateway-7876bfcb74-lwn5j Add eth0 [10.134.0.20/23] from ovn-kubernetes 11m Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7876bfcb74 Created pod: istiod-openshift-gateway-7876bfcb74-lwn5j 11m Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7876bfcb74 from 0 to 1 11m Normal Pulling pod/istiod-openshift-gateway-7876bfcb74-lwn5j Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 11m Normal Started pod/istiod-openshift-gateway-7876bfcb74-lwn5j Container started 11m Normal Created pod/istiod-openshift-gateway-7876bfcb74-lwn5j Container created 11m Normal Pulled pod/istiod-openshift-gateway-7876bfcb74-lwn5j Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.422s (3.422s including waiting). Image size: 185476477 bytes. 11m Normal EnsuringLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensuring load balancer 11m Warning Unhealthy pod/istiod-openshift-gateway-7876bfcb74-lwn5j Readiness probe failed: HTTP probe failed with statuscode: 503 11m Normal ScalingReplicaSet deployment/openshift-ai-inference-openshift-ai-inference Scaled up replica set openshift-ai-inference-openshift-ai-inference-85f45cdb7 from 0 to 1 11m Normal SuccessfulCreate replicaset/openshift-ai-inference-openshift-ai-inference-85f45cdb7 Created pod: openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf 11m Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7d64fcddb7 from 0 to 1 11m Warning ProbeError pod/istiod-openshift-gateway-7876bfcb74-lwn5j Readiness probe error: HTTP probe failed with statuscode: 503... 11m Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-skgsr 10m Normal AddedInterface pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Add eth0 [10.132.0.31/23] from ovn-kubernetes 10m Normal Pulling pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 10m Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Add eth0 [10.133.0.20/23] from ovn-kubernetes 10m Normal Pulling pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 10m Normal EnsuredLoadBalancer service/openshift-ai-inference-openshift-ai-inference Ensured load balancer 10m Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 8.393s (8.393s including waiting). Image size: 183555333 bytes. 10m Normal Pulled pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 9.766s (9.766s including waiting). Image size: 183555333 bytes. 10m Normal Created pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Container created 10m Normal Started pod/openshift-ai-inference-openshift-ai-inference-85f45cdb7-xc6kf Container started 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 10m Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedGetResourceMetric horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 10m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/openshift-ai-inference-openshift-ai-inference invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m45s Normal SuccessfulCreate replicaset/kube-auth-proxy-5bd469f65f Created pod: kube-auth-proxy-5bd469f65f-x7t8v 9m45s Normal SuccessfulCreate replicaset/kube-auth-proxy-5bd469f65f Created pod: kube-auth-proxy-5bd469f65f-4x7zx 9m45s Normal ScalingReplicaSet deployment/kube-auth-proxy Scaled up replica set kube-auth-proxy-5bd469f65f from 0 to 2 9m44s Normal Started pod/data-science-gateway-data-science-gateway-class-597d745469j8tv2 Container started 9m44s Normal AddedInterface pod/data-science-gateway-data-science-gateway-class-597d745469j8tv2 Add eth0 [10.133.0.29/23] from ovn-kubernetes 9m44s Normal Pulling pod/kube-auth-proxy-5bd469f65f-4x7zx Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 9m44s Normal AddedInterface pod/kube-auth-proxy-5bd469f65f-4x7zx Add eth0 [10.134.0.23/23] from ovn-kubernetes 9m44s Normal SuccessfulCreate replicaset/data-science-gateway-data-science-gateway-class-597d745469 Created pod: data-science-gateway-data-science-gateway-class-597d745469j8tv2 9m44s Normal Pulled pod/data-science-gateway-data-science-gateway-class-597d745469j8tv2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 9m44s Normal ScalingReplicaSet deployment/data-science-gateway-data-science-gateway-class Scaled up replica set data-science-gateway-data-science-gateway-class-597d745469 from 0 to 1 9m44s Normal Created pod/data-science-gateway-data-science-gateway-class-597d745469j8tv2 Container created 9m44s Normal AddedInterface pod/kube-auth-proxy-5bd469f65f-x7t8v Add eth0 [10.132.0.41/23] from ovn-kubernetes 9m44s Normal Pulling pod/kube-auth-proxy-5bd469f65f-x7t8v Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" 9m41s Normal Pulled pod/kube-auth-proxy-5bd469f65f-4x7zx Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.865s (2.865s including waiting). Image size: 188904677 bytes. 9m41s Normal Created pod/kube-auth-proxy-5bd469f65f-x7t8v Container created 9m41s Normal Started pod/kube-auth-proxy-5bd469f65f-x7t8v Container started 9m41s Normal Pulled pod/kube-auth-proxy-5bd469f65f-x7t8v Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:d90930380151490d5516eeb95840eecc743e9005fc02733dec6fe79e1a00e035" in 2.967s (2.967s including waiting). Image size: 188904677 bytes. 9m41s Normal Started pod/kube-auth-proxy-5bd469f65f-4x7zx Container started 9m41s Normal Created pod/kube-auth-proxy-5bd469f65f-4x7zx Container created 9m30s Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m30s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m27s Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m27s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 9m14s Warning FailedGetResourceMetric horizontalpodautoscaler/kube-auth-proxy failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m14s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/kube-auth-proxy invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m12s Warning FailedGetResourceMetric horizontalpodautoscaler/data-science-gateway-data-science-gateway-class failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 9m12s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/data-science-gateway-data-science-gateway-class invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 8m45s Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" already present on machine and can be accessed by the pod 8m45s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 0 to 1 8m45s Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Container created 8m45s Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-82hvc 8m45s Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Add eth0 [10.134.0.26/23] from ovn-kubernetes 8m45s Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Container started 8m44s Warning ProbeError pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Readiness probe error: HTTP probe failed with statuscode: 503... 8m44s Warning Unhealthy pod/istiod-openshift-gateway-7ddd9cfdc8-82hvc Readiness probe failed: HTTP probe failed with statuscode: 503 8m43s Normal Killing pod/istiod-openshift-gateway-7876bfcb74-lwn5j Stopping container discovery 8m43s Normal SuccessfulDelete replicaset/istiod-openshift-gateway-7876bfcb74 Deleted pod: istiod-openshift-gateway-7876bfcb74-lwn5j 8m43s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled down replica set istiod-openshift-gateway-7876bfcb74 from 1 to 0 8m8s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 8m7s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Container started 8m7s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Container created 8m5s Warning FailedGetResourceMetric horizontalpodautoscaler/istiod-openshift-gateway failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m5s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/istiod-openshift-gateway invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m1s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Startup probe failed: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused 8m1s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-skgsr Startup probe error: Get "http://10.133.0.20:15021/healthz/ready": dial tcp 10.133.0.20:15021: connect: connection refused... 8m Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 8m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 7m45s Warning FailedGetResourceMetric horizontalpodautoscaler/maas-default-gateway-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 7m45s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/maas-default-gateway-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 6m29s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7d64fcddb7 from 1 to 4 6m29s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 6m29s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Container started 6m29s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 6m29s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-xrqpp 6m29s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Container created 6m29s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Add eth0 [10.134.0.32/23] from ovn-kubernetes 6m29s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m29s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Add eth0 [10.132.0.55/23] from ovn-kubernetes 6m29s Normal Pulling pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 6m29s Normal Pulling pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" 6m29s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Add eth0 [10.134.0.31/23] from ovn-kubernetes 6m29s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 6m25s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Container created 6m25s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Container started 6m25s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Container created 6m25s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.681s (3.681s including waiting). Image size: 183555333 bytes. 6m25s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" in 3.609s (3.609s including waiting). Image size: 183555333 bytes. 6m25s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Container started 6m21s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Startup probe error: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused... 6m21s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-4tpt8 Startup probe failed: Get "http://10.132.0.55:15021/healthz/ready": dial tcp 10.132.0.55:15021: connect: connection refused 6m18s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Startup probe error: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused... 6m18s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-xrqpp Startup probe failed: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused 6m17s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Startup probe error: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused... 6m17s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-wjbj2 Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused 6m14s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Add eth0 [10.132.0.57/23] from ovn-kubernetes 6m14s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Container started 6m14s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Container created 6m14s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m14s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m14s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Add eth0 [10.133.0.39/23] from ovn-kubernetes 6m14s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 6m14s Warning FailedCreate replicaset/payload-processing-7bb9489676 Error creating: pods "payload-processing-7bb9489676-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 6m14s Warning FailedCreate replicaset/payload-pre-processing-7b98bb678c Error creating: pods "payload-pre-processing-7b98bb678c-" is forbidden: error looking up service account openshift-ingress/payload-processing: serviceaccount "payload-processing" not found 6m14s Normal ScalingReplicaSet deployment/payload-pre-processing Scaled up replica set payload-pre-processing-7b98bb678c from 0 to 1 6m14s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-96lb9 6m14s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-n5x92 6m14s Normal SuccessfulCreate replicaset/maas-default-gateway-openshift-default-7d64fcddb7 Created pod: maas-default-gateway-openshift-default-7d64fcddb7-gnswf 6m14s Normal SuccessfulCreate replicaset/payload-pre-processing-7b98bb678c Created pod: payload-pre-processing-7b98bb678c-lqlbm 6m14s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Add eth0 [10.134.0.34/23] from ovn-kubernetes 6m14s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m14s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Container created 6m14s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Container created 6m14s Normal Pulled pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 6m14s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Container started 6m14s Normal ScalingReplicaSet deployment/payload-processing Scaled up replica set payload-processing-7bb9489676 from 0 to 1 6m14s Normal Created pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Container created 6m14s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Container started 6m14s Normal AddedInterface pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Add eth0 [10.133.0.40/23] from ovn-kubernetes 6m14s Normal SuccessfulRescale horizontalpodautoscaler/maas-default-gateway-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 6m14s Normal ScalingReplicaSet deployment/maas-default-gateway-openshift-default Scaled up replica set maas-default-gateway-openshift-default-7d64fcddb7 from 4 to 8 6m14s Normal Started pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Container started 6m13s Normal SuccessfulCreate replicaset/payload-processing-7bb9489676 Created pod: payload-processing-7bb9489676-sv8xh 6m13s Normal AddedInterface pod/payload-processing-7bb9489676-sv8xh Add eth0 [10.132.0.59/23] from ovn-kubernetes 6m13s Normal Pulling pod/payload-pre-processing-7b98bb678c-lqlbm Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 6m13s Normal AddedInterface pod/payload-pre-processing-7b98bb678c-lqlbm Add eth0 [10.132.0.58/23] from ovn-kubernetes 6m13s Normal Pulling pod/payload-processing-7bb9489676-sv8xh Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 6m9s Normal Created pod/payload-processing-7bb9489676-sv8xh Container created 6m9s Normal Created pod/payload-pre-processing-7b98bb678c-lqlbm Container created 6m9s Normal Started pod/payload-pre-processing-7b98bb678c-lqlbm Container started 6m9s Normal Pulled pod/payload-processing-7bb9489676-sv8xh Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.844s (3.844s including waiting). Image size: 125353963 bytes. 6m9s Normal Started pod/payload-processing-7bb9489676-sv8xh Container started 6m9s Normal Pulled pod/payload-pre-processing-7b98bb678c-lqlbm Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.834s (3.834s including waiting). Image size: 125353963 bytes. 6m7s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Startup probe failed: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused 6m7s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Startup probe error: Get "http://10.134.0.34:15021/healthz/ready": dial tcp 10.134.0.34:15021: connect: connection refused... 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": dial tcp 10.133.0.39:15021: connect: connection refused 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused... 6m6s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": dial tcp 10.133.0.40:15021: connect: connection refused 6m6s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": dial tcp 10.132.0.57:15021: connect: connection refused... 6m3s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Startup probe error: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m3s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Startup probe error: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m3s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-n5x92 Startup probe failed: Get "http://10.132.0.57:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m3s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-kkgr2 Startup probe failed: Get "http://10.133.0.39:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m3s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Startup probe error: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 6m3s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-96lb9 Startup probe failed: Get "http://10.134.0.34:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m2s Warning Unhealthy pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Startup probe failed: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 6m2s Warning ProbeError pod/maas-default-gateway-openshift-default-7d64fcddb7-gnswf Startup probe error: Get "http://10.133.0.40:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m22s Normal AddedLabel gateway/e2e-worker-w3-998997 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w3-998997 5m21s Normal AddedLabel gateway/e2e-worker-w5-56a44f Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w5-56a44f 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w3-998997-openshift-default-768df74449 Created pod: e2e-worker-w3-998997-openshift-default-768df74449-8f6fh 5m21s Normal AddedInterface pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Add eth0 [10.134.0.38/23] from ovn-kubernetes 5m21s Normal AddedLabel gateway/e2e-worker-w4-64c20d Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w4-64c20d 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f Created pod: e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w0-4afe0e-openshift-default Scaled up replica set e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f from 0 to 1 5m21s Normal AddedLabel gateway/e2e-worker-w0-4afe0e Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w0-4afe0e 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-f866m 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w5-56a44f-openshift-default Scaled up replica set e2e-worker-w5-56a44f-openshift-default-584894595f from 0 to 1 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w3-998997-openshift-default Scaled up replica set e2e-worker-w3-998997-openshift-default-768df74449 from 0 to 1 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w2-78ca37-openshift-default-6d9b468c59 Created pod: e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w1-36730f-openshift-default-84ffb6c657 Created pod: e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp 5m21s Normal AddedInterface pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Add eth0 [10.134.0.36/23] from ovn-kubernetes 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w4-64c20d-openshift-default Scaled up replica set e2e-worker-w4-64c20d-openshift-default-d97b6c887 from 0 to 1 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w1-36730f-openshift-default Scaled up replica set e2e-worker-w1-36730f-openshift-default-84ffb6c657 from 0 to 1 5m21s Normal AddedLabel gateway/e2e-worker-w2-78ca37 Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w2-78ca37 5m21s Normal SuccessfulCreate replicaset/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 Created pod: e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w6-6a9cdb-openshift-default Scaled up replica set e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 from 0 to 1 5m21s Normal AddedLabel gateway/e2e-worker-w1-36730f Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w1-36730f 5m21s Normal ScalingReplicaSet deployment/e2e-worker-w2-78ca37-openshift-default Scaled up replica set e2e-worker-w2-78ca37-openshift-default-6d9b468c59 from 0 to 1 5m21s Normal AddedLabel gateway/e2e-worker-w6-6a9cdb Added label istio.io/rev=openshift-gateway to gateway e2e-worker-w6-6a9cdb 5m20s Normal AddedInterface pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Add eth0 [10.133.0.42/23] from ovn-kubernetes 5m20s Normal AddedInterface pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Add eth0 [10.134.0.39/23] from ovn-kubernetes 5m20s Normal AddedInterface pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Add eth0 [10.134.0.40/23] from ovn-kubernetes 5m20s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Add eth0 [10.133.0.41/23] from ovn-kubernetes 5m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Add eth0 [10.134.0.37/23] from ovn-kubernetes 5m17s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f Error creating: pods "payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-998997: serviceaccount "payload-processing-e2e-worker-w3-998997" not found 5m17s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w3-998997 Scaled up replica set payload-processing-e2e-worker-w3-998997-f84f87f66 from 0 to 1 5m17s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w3-998997 Scaled up replica set payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f from 0 to 1 5m17s Warning FailedCreate replicaset/payload-processing-e2e-worker-w3-998997-f84f87f66 Error creating: pods "payload-processing-e2e-worker-w3-998997-f84f87f66-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w3-998997: serviceaccount "payload-processing-e2e-worker-w3-998997" not found 5m17s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w3-998997-f84f87f66 Created pod: payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn 5m17s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f Created pod: payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs 5m16s Normal Started pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Container started 5m16s Normal Created pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Container created 5m16s Normal Started pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Container started 5m16s Normal Pulled pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m16s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Add eth0 [10.132.0.60/23] from ovn-kubernetes 5m16s Normal Created pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Container created 5m16s Normal Pulled pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m16s Normal AddedInterface pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Add eth0 [10.132.0.61/23] from ovn-kubernetes 5m15s Normal AddedInterface pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Add eth0 [10.132.0.63/23] from ovn-kubernetes 5m15s Normal Pulled pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m15s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff Error creating: pods "payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-4afe0e: serviceaccount "payload-processing-e2e-worker-w0-4afe0e" not found 5m15s Normal Started pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Container started 5m15s Warning FailedCreate replicaset/payload-processing-e2e-worker-w0-4afe0e-6f5f884484 Error creating: pods "payload-processing-e2e-worker-w0-4afe0e-6f5f884484-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w0-4afe0e: serviceaccount "payload-processing-e2e-worker-w0-4afe0e" not found 5m15s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Add eth0 [10.132.0.62/23] from ovn-kubernetes 5m15s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff Created pod: payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st 5m15s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w0-4afe0e Scaled up replica set payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff from 0 to 1 5m15s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w0-4afe0e-6f5f884484 Created pod: payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 5m15s Normal Created pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Container created 5m15s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w0-4afe0e Scaled up replica set payload-processing-e2e-worker-w0-4afe0e-6f5f884484 from 0 to 1 5m15s Normal Pulled pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m14s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Add eth0 [10.132.0.65/23] from ovn-kubernetes 5m14s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w5-56a44f Scaled up replica set payload-processing-e2e-worker-w5-56a44f-576768df8d from 0 to 1 5m14s Normal Pulled pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m14s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w5-56a44f-576768df8d Created pod: payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l 5m14s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc Error creating: pods "payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-56a44f: serviceaccount "payload-processing-e2e-worker-w5-56a44f" not found 5m14s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc Created pod: payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp 5m14s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w5-56a44f Scaled up replica set payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc from 0 to 1 5m14s Warning FailedCreate replicaset/payload-processing-e2e-worker-w5-56a44f-576768df8d Error creating: pods "payload-processing-e2e-worker-w5-56a44f-576768df8d-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w5-56a44f: serviceaccount "payload-processing-e2e-worker-w5-56a44f" not found 5m14s Normal AddedInterface pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Add eth0 [10.133.0.43/23] from ovn-kubernetes 5m14s Normal Started pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Container started 5m14s Normal Created pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Container created 5m13s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c Created pod: payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s 5m13s Normal AddedInterface pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Add eth0 [10.132.0.67/23] from ovn-kubernetes 5m13s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c Error creating: pods "payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb: serviceaccount "payload-processing-e2e-worker-w6-6a9cdb" not found 5m13s Normal Created pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Container created 5m13s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w6-6a9cdb Scaled up replica set payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c from 0 to 1 5m13s Normal Pulled pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m13s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w6-6a9cdb Scaled up replica set payload-processing-e2e-worker-w6-6a9cdb-5446d884cb from 0 to 1 5m13s Normal Started pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Container started 5m13s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Add eth0 [10.133.0.44/23] from ovn-kubernetes 5m13s Normal Pulling pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 5m13s Normal Pulling pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 5m13s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb Created pod: payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 5m13s Warning FailedCreate replicaset/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb Error creating: pods "payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w6-6a9cdb: serviceaccount "payload-processing-e2e-worker-w6-6a9cdb" not found 5m13s Normal Created pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Container created 5m13s Normal Started pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Container started 5m12s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w2-78ca37 Scaled up replica set payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f from 0 to 1 5m12s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f Created pod: payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 5m12s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w2-78ca37 Scaled up replica set payload-processing-e2e-worker-w2-78ca37-5dd495d496 from 0 to 1 5m12s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w2-78ca37-5dd495d496 Created pod: payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d 5m12s Warning FailedCreate replicaset/payload-processing-e2e-worker-w2-78ca37-5dd495d496 Error creating: pods "payload-processing-e2e-worker-w2-78ca37-5dd495d496-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-78ca37: serviceaccount "payload-processing-e2e-worker-w2-78ca37" not found 5m12s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w1-36730f Scaled up replica set payload-processing-e2e-worker-w1-36730f-7cfcf6d874 from 0 to 1 5m12s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795 Error creating: pods "payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-36730f: serviceaccount "payload-processing-e2e-worker-w1-36730f" not found 5m12s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w1-36730f-7cfcf6d874 Created pod: payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg 5m12s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795 Created pod: payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s 5m12s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w1-36730f Scaled up replica set payload-pre-processing-e2e-worker-w1-36730f-6df8c45795 from 0 to 1 5m12s Warning FailedCreate replicaset/payload-processing-e2e-worker-w1-36730f-7cfcf6d874 Error creating: pods "payload-processing-e2e-worker-w1-36730f-7cfcf6d874-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w1-36730f: serviceaccount "payload-processing-e2e-worker-w1-36730f" not found 5m12s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f Error creating: pods "payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w2-78ca37: serviceaccount "payload-processing-e2e-worker-w2-78ca37" not found 5m12s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Add eth0 [10.132.0.68/23] from ovn-kubernetes 5m11s Warning FailedCreate replicaset/payload-processing-e2e-worker-w4-64c20d-574b8745d4 Error creating: pods "payload-processing-e2e-worker-w4-64c20d-574b8745d4-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-64c20d: serviceaccount "payload-processing-e2e-worker-w4-64c20d" not found 5m11s Warning FailedCreate replicaset/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5 Error creating: pods "payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-" is forbidden: error looking up service account openshift-ingress/payload-processing-e2e-worker-w4-64c20d: serviceaccount "payload-processing-e2e-worker-w4-64c20d" not found 5m11s Normal Pulled pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m11s Normal AddedInterface pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Add eth0 [10.132.0.69/23] from ovn-kubernetes 5m11s Normal Pulled pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m11s Normal Created pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Container created 5m11s Normal SuccessfulCreate replicaset/payload-processing-e2e-worker-w4-64c20d-574b8745d4 Created pod: payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r 5m11s Normal Started pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Container started 5m11s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Add eth0 [10.134.0.42/23] from ovn-kubernetes 5m11s Normal Pulling pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 5m11s Normal Created pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Container created 5m11s Normal Started pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Container started 5m11s Normal SuccessfulCreate replicaset/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5 Created pod: payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd 5m11s Normal Pulling pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Pulling image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" 5m11s Normal AddedInterface pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Add eth0 [10.134.0.41/23] from ovn-kubernetes 5m11s Normal ScalingReplicaSet deployment/payload-pre-processing-e2e-worker-w4-64c20d Scaled up replica set payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5 from 0 to 1 5m11s Normal ScalingReplicaSet deployment/payload-processing-e2e-worker-w4-64c20d Scaled up replica set payload-processing-e2e-worker-w4-64c20d-574b8745d4 from 0 to 1 5m10s Normal Pulled pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.034s (3.034s including waiting). Image size: 125353963 bytes. 5m10s Normal Created pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Container created 5m10s Normal Pulled pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m10s Normal AddedInterface pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Add eth0 [10.133.0.46/23] from ovn-kubernetes 5m10s Normal Pulled pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 3.13s (3.13s including waiting). Image size: 125353963 bytes. 5m10s Normal Created pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Container created 5m10s Normal Pulled pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Container image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" already present on machine and can be accessed by the pod 5m10s Normal Started pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Container started 5m10s Normal Created pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Container created 5m10s Normal Created pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Container created 5m10s Normal AddedInterface pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Add eth0 [10.133.0.45/23] from ovn-kubernetes 5m10s Normal Started pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Container started 5m10s Normal Started pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Container started 5m9s Normal Started pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Container started 5m5s Normal Created pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Container created 5m5s Normal Pulled pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 6.446s (6.446s including waiting). Image size: 125353963 bytes. 5m5s Normal Started pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Container started 5m5s Normal Pulled pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Successfully pulled image "quay.io/opendatahub/odh-praxis-extproc:odh-stable" in 6.651s (6.651s including waiting). Image size: 125353963 bytes. 5m4s Normal Created pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Container created 5m4s Normal Started pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Container started 5m3s Warning ProbeError pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Readiness probe error: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m3s Warning Unhealthy pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Readiness probe failed: Get "http://10.134.0.36:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5m2s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 5m2s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 4m58s Normal Created pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Container created 4m58s Normal Pulled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m58s Normal Created pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Container created 4m58s Normal Started pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Container started 4m58s Normal Pulled pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m58s Normal Started pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Container started 4m56s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m56s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Container created 4m56s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Container started 4m55s Normal Started pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Container started 4m55s Normal Created pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Container created 4m55s Normal Pulled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m54s Normal Created pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Container created 4m54s Normal Pulled pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m54s Normal Started pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Container started 4m53s Normal Pulled pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m52s Normal Created pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Container created 4m52s Normal Started pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Container started 4m52s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Startup probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused... 4m52s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Startup probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 4m52s Warning Unhealthy pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Startup probe failed: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused 4m52s Warning ProbeError pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Startup probe error: Get "http://10.134.0.36:15021/healthz/ready": dial tcp 10.134.0.36:15021: connect: connection refused... 4m51s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Startup probe error: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused... 4m51s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Startup probe failed: Get "http://10.133.0.41:15021/healthz/ready": dial tcp 10.133.0.41:15021: connect: connection refused 4m51s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-36730f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m51s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-36730f-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m51s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-998997-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m51s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-998997-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m51s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m51s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m49s Warning ProbeError pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Startup probe error: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused... 4m49s Warning Unhealthy pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Startup probe failed: Get "http://10.133.0.42:15021/healthz/ready": dial tcp 10.133.0.42:15021: connect: connection refused 4m47s Warning Unhealthy pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Startup probe failed: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused 4m47s Warning ProbeError pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Startup probe error: Get "http://10.134.0.40:15021/healthz/ready": dial tcp 10.134.0.40:15021: connect: connection refused... 4m45s Warning Unhealthy pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused 4m45s Warning ProbeError pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Startup probe error: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused... 4m43s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m43s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Container created 4m43s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Container started 4m38s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Startup probe error: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused... 4m38s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Startup probe failed: Get "http://10.134.0.37:15021/healthz/ready": dial tcp 10.134.0.37:15021: connect: connection refused 4m36s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m36s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m36s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m36s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m35s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-78ca37-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m35s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-78ca37-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w5-56a44f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w5-56a44f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w1-36730f-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m21s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m21s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m21s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w1-36730f-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m20s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w2-78ca37-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m20s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w2-78ca37-openshift-default failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) 4m6s Normal ScalingReplicaSet deployment/e2e-worker-w6-6a9cdb-openshift-default Scaled up replica set e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 from 1 to 2 4m6s Normal SuccessfulCreate replicaset/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 Created pod: e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h 4m6s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs 4m6s Normal SuccessfulCreate replicaset/e2e-worker-w1-36730f-openshift-default-84ffb6c657 Created pod: e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg 4m6s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m6s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 4m6s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w1-36730f-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 4m6s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m6s Warning FailedGetResourceMetric horizontalpodautoscaler/e2e-worker-w3-998997-openshift-default failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m6s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-56a44f-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 4m6s Normal ScalingReplicaSet deployment/e2e-worker-w5-56a44f-openshift-default Scaled up replica set e2e-worker-w5-56a44f-openshift-default-584894595f from 1 to 2 4m6s Normal ScalingReplicaSet deployment/e2e-worker-w1-36730f-openshift-default Scaled up replica set e2e-worker-w1-36730f-openshift-default-84ffb6c657 from 1 to 2 4m6s Warning FailedComputeMetricsReplicas horizontalpodautoscaler/e2e-worker-w3-998997-openshift-default invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API 4m5s Normal Pulled pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m5s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Container started 4m5s Normal AddedInterface pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Add eth0 [10.133.0.55/23] from ovn-kubernetes 4m5s Normal Started pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Container started 4m5s Normal Created pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Container created 4m5s Normal Pulled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m5s Normal AddedInterface pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Add eth0 [10.134.0.50/23] from ovn-kubernetes 4m5s Normal Created pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Container created 4m5s Normal Started pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Container started 4m5s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Add eth0 [10.134.0.49/23] from ovn-kubernetes 4m5s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 4m5s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Container created 4m Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Startup probe error: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused... 4m Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused 3m59s Warning ProbeError pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Startup probe error: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused... 3m59s Warning Unhealthy pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Startup probe failed: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused 3m59s Warning ProbeError pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Startup probe error: Get "http://10.134.0.50:15021/healthz/ready": dial tcp 10.134.0.50:15021: connect: connection refused... 3m59s Warning Unhealthy pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Startup probe failed: Get "http://10.133.0.55:15021/healthz/ready": dial tcp 10.133.0.55:15021: connect: connection refused 3m51s Normal SuccessfulCreate replicaset/e2e-worker-w3-998997-openshift-default-768df74449 Created pod: e2e-worker-w3-998997-openshift-default-768df74449-6f2pc 3m51s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w3-998997-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 3m51s Normal AddedInterface pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Add eth0 [10.133.0.59/23] from ovn-kubernetes 3m51s Normal Pulled pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m51s Normal ScalingReplicaSet deployment/e2e-worker-w3-998997-openshift-default Scaled up replica set e2e-worker-w3-998997-openshift-default-768df74449 from 1 to 2 3m50s Normal Created pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Container created 3m50s Normal Started pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Container started 3m45s Warning ProbeError pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused... 3m45s Warning Unhealthy pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": dial tcp 10.133.0.59:15021: connect: connection refused 3m43s Warning ProbeError pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Startup probe error: Get "http://10.133.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 3m43s Warning Unhealthy pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Startup probe failed: Get "http://10.133.0.59:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 3m35s Normal Pulled pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m35s Normal Started pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Container started 3m35s Normal SuccessfulCreate replicaset/e2e-worker-w2-78ca37-openshift-default-6d9b468c59 Created pod: e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w 3m35s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w2-78ca37-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 3m35s Normal ScalingReplicaSet deployment/e2e-worker-w2-78ca37-openshift-default Scaled up replica set e2e-worker-w2-78ca37-openshift-default-6d9b468c59 from 1 to 2 3m35s Normal Created pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Container created 3m35s Normal AddedInterface pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Add eth0 [10.133.0.60/23] from ovn-kubernetes 3m28s Warning ProbeError pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Startup probe error: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused... 3m28s Warning Unhealthy pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Startup probe failed: Get "http://10.133.0.60:15021/healthz/ready": dial tcp 10.133.0.60:15021: connect: connection refused 3m21s Normal ScalingReplicaSet deployment/e2e-worker-w4-64c20d-openshift-default Scaled up replica set e2e-worker-w4-64c20d-openshift-default-d97b6c887 from 1 to 3 3m21s Normal ScalingReplicaSet deployment/e2e-worker-w0-4afe0e-openshift-default Scaled up replica set e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f from 1 to 2 3m21s Normal SuccessfulCreate replicaset/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f Created pod: e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd 3m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 3m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf 3m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default New size: 2; reason: cpu resource utilization (percentage of request) above target 3m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv 3m20s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Container created 3m20s Normal Created pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Container created 3m20s Normal Pulled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m20s Normal AddedInterface pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Add eth0 [10.133.0.62/23] from ovn-kubernetes 3m20s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Container started 3m20s Normal Started pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Container started 3m20s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Container created 3m20s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Add eth0 [10.133.0.61/23] from ovn-kubernetes 3m20s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Container started 3m20s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Add eth0 [10.132.0.75/23] from ovn-kubernetes 3m14s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Startup probe error: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused... 3m14s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Startup probe failed: Get "http://10.132.0.75:15021/healthz/ready": dial tcp 10.132.0.75:15021: connect: connection refused 3m14s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Startup probe failed: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused 3m14s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Startup probe error: Get "http://10.133.0.61:15021/healthz/ready": dial tcp 10.133.0.61:15021: connect: connection refused... 3m14s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Startup probe failed: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused 3m14s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Startup probe error: Get "http://10.133.0.62:15021/healthz/ready": dial tcp 10.133.0.62:15021: connect: connection refused... 3m5s Normal Created pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Container created 3m5s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w6-6a9cdb-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 3m5s Normal ScalingReplicaSet deployment/e2e-worker-w6-6a9cdb-openshift-default Scaled up replica set e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 from 2 to 3 3m5s Normal SuccessfulCreate replicaset/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555 Created pod: e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl 3m5s Normal AddedInterface pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Add eth0 [10.132.0.78/23] from ovn-kubernetes 3m5s Normal Pulled pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 3m5s Normal Started pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Container started 2m59s Warning ProbeError pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Startup probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 2m59s Warning Unhealthy pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Startup probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 2m53s Normal Killing pod/payload-pre-processing-e2e-worker-w3-998997-65cfb47f7f-wtzrs Stopping container payload-pre-processing 2m53s Normal Killing pod/payload-processing-e2e-worker-w3-998997-f84f87f66-mg8dn Stopping container payload-processing 2m49s Normal Killing pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Stopping container payload-pre-processing 2m49s Normal Killing pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Stopping container payload-processing 2m46s Normal Killing pod/payload-processing-e2e-worker-w6-6a9cdb-5446d884cb-k4ts2 Stopping container payload-processing 2m46s Normal Killing pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Stopping container payload-pre-processing 2m45s Normal Killing pod/payload-pre-processing-e2e-worker-w2-78ca37-cc6b6c59f-tdgv7 Stopping container payload-pre-processing 2m45s Normal Killing pod/payload-processing-e2e-worker-w2-78ca37-5dd495d496-cv49d Stopping container payload-processing 2m36s Normal ScalingReplicaSet deployment/e2e-worker-w5-56a44f-openshift-default Scaled up replica set e2e-worker-w5-56a44f-openshift-default-584894595f from 2 to 3 2m36s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 2m36s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-56a44f-openshift-default New size: 3; reason: cpu resource utilization (percentage of request) above target 2m35s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m35s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Container started 2m35s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Container created 2m35s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Add eth0 [10.132.0.81/23] from ovn-kubernetes 2m33s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w1-36730f-6df8c45795-sfr6s Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 2m28s Warning Unhealthy pod/payload-processing-e2e-worker-w1-36730f-7cfcf6d874-c6sgg Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 2m28s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Startup probe failed: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused 2m28s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Startup probe error: Get "http://10.132.0.81:15021/healthz/ready": dial tcp 10.132.0.81:15021: connect: connection refused... 2m26s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w6-6a9cdb-6bd44cf46c-nwp8s Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 2m25s Normal Killing pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Stopping container payload-pre-processing 2m25s Normal Killing pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Stopping container payload-processing 2m22s Normal Killing pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Stopping container istio-proxy 2m22s Normal Killing pod/e2e-worker-w3-998997-openshift-default-768df74449-6f2pc Stopping container istio-proxy 2m21s Normal ScalingReplicaSet deployment/e2e-worker-w5-56a44f-openshift-default Scaled up replica set e2e-worker-w5-56a44f-openshift-default-584894595f from 3 to 6 2m21s Warning ProbeError pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Readiness probe error: HTTP probe failed with statuscode: 503... 2m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w0-4afe0e-openshift-default New size: 4; reason: cpu resource utilization (percentage of request) above target 2m21s Warning Unhealthy pod/e2e-worker-w3-998997-openshift-default-768df74449-8f6fh Readiness probe failed: HTTP probe failed with statuscode: 503 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c 2m21s Normal ScalingReplicaSet deployment/e2e-worker-w0-4afe0e-openshift-default Scaled up replica set e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f from 2 to 4 2m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w5-56a44f-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f Created pod: e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 2m21s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default New size: 6; reason: cpu resource utilization (percentage of request) above target 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w5-56a44f-openshift-default-584894595f Created pod: e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s 2m21s Normal ScalingReplicaSet deployment/e2e-worker-w4-64c20d-openshift-default Scaled up replica set e2e-worker-w4-64c20d-openshift-default-d97b6c887 from 3 to 6 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f Created pod: e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd 2m21s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb 2m20s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Container created 2m20s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal AddedInterface pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Add eth0 [10.134.0.73/23] from ovn-kubernetes 2m20s Normal Pulled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Add eth0 [10.134.0.72/23] from ovn-kubernetes 2m20s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Container started 2m20s Normal Created pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Container created 2m20s Normal Started pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Container started 2m20s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Container started 2m20s Normal AddedInterface pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Add eth0 [10.132.0.83/23] from ovn-kubernetes 2m20s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Container created 2m20s Normal Pulled pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal Created pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Container created 2m20s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Add eth0 [10.134.0.74/23] from ovn-kubernetes 2m20s Normal Started pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Container started 2m20s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Container started 2m20s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Container started 2m20s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Container created 2m20s Normal Started pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Container started 2m20s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Add eth0 [10.132.0.84/23] from ovn-kubernetes 2m20s Normal AddedInterface pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Add eth0 [10.133.0.64/23] from ovn-kubernetes 2m20s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Container created 2m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Add eth0 [10.132.0.82/23] from ovn-kubernetes 2m20s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Container created 2m20s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Container started 2m20s Normal Pulled pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal Created pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Container created 2m20s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m20s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Add eth0 [10.133.0.63/23] from ovn-kubernetes 2m18s Normal Killing pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Stopping container istio-proxy 2m18s Normal Killing pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-clh9w Stopping container istio-proxy 2m18s Normal Killing pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Stopping container istio-proxy 2m18s Normal Killing pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Stopping container istio-proxy 2m17s Normal Killing pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Stopping container istio-proxy 2m17s Normal Killing pod/payload-processing-e2e-worker-w4-64c20d-574b8745d4-m8m7r Stopping container payload-processing 2m17s Normal Killing pod/payload-pre-processing-e2e-worker-w4-64c20d-5c65bd88c5-hrbnd Stopping container payload-pre-processing 2m14s Warning Unhealthy pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Readiness probe failed: HTTP probe failed with statuscode: 503 2m14s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Startup probe failed: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused 2m14s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Startup probe error: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused... 2m14s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Startup probe failed: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused 2m14s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Startup probe error: Get "http://10.132.0.82:15021/healthz/ready": dial tcp 10.132.0.82:15021: connect: connection refused... 2m14s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Startup probe failed: Get "http://10.133.0.63:15021/healthz/ready": dial tcp 10.133.0.63:15021: connect: connection refused 2m14s Warning ProbeError pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-ktksp Readiness probe error: HTTP probe failed with statuscode: 503... 2m14s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Startup probe error: Get "http://10.132.0.83:15021/healthz/ready": dial tcp 10.132.0.83:15021: connect: connection refused... 2m13s Warning ProbeError pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Readiness probe error: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused... 2m13s Normal Killing pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-sf4jv Stopping container istio-proxy 2m13s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Startup probe error: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused... 2m13s Warning Unhealthy pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Readiness probe failed: HTTP probe failed with statuscode: 503 2m13s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Startup probe failed: Get "http://10.133.0.64:15021/healthz/ready": dial tcp 10.133.0.64:15021: connect: connection refused 2m13s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Startup probe failed: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused 2m13s Warning Unhealthy pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Readiness probe failed: HTTP probe failed with statuscode: 503 2m13s Warning Unhealthy pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-snvnl Readiness probe failed: Get "http://10.132.0.78:15021/healthz/ready": dial tcp 10.132.0.78:15021: connect: connection refused 2m13s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Startup probe error: Get "http://10.132.0.84:15021/healthz/ready": dial tcp 10.132.0.84:15021: connect: connection refused... 2m13s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Startup probe failed: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused 2m13s Normal Killing pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Stopping container istio-proxy 2m13s Warning ProbeError pod/e2e-worker-w1-36730f-openshift-default-84ffb6c657-hjhdg Readiness probe error: HTTP probe failed with statuscode: 503... 2m13s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Startup probe error: Get "http://10.134.0.72:15021/healthz/ready": dial tcp 10.134.0.72:15021: connect: connection refused... 2m13s Warning ProbeError pod/e2e-worker-w6-6a9cdb-openshift-default-6779d6d555-j4t9h Readiness probe error: HTTP probe failed with statuscode: 503... 2m12s Warning Unhealthy pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Readiness probe failed: HTTP probe failed with statuscode: 503 2m12s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Startup probe failed: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused 2m12s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Startup probe error: Get "http://10.134.0.73:15021/healthz/ready": dial tcp 10.134.0.73:15021: connect: connection refused... 2m12s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Startup probe failed: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused 2m12s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Startup probe error: Get "http://10.134.0.74:15021/healthz/ready": dial tcp 10.134.0.74:15021: connect: connection refused... 2m12s Warning ProbeError pod/e2e-worker-w2-78ca37-openshift-default-6d9b468c59-dkf7w Readiness probe error: HTTP probe failed with statuscode: 503... 2m8s Warning Unhealthy pod/payload-processing-e2e-worker-w5-56a44f-576768df8d-ddt9l Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 2m6s Normal SuccessfulRescale horizontalpodautoscaler/e2e-worker-w4-64c20d-openshift-default New size: 8; reason: cpu resource utilization (percentage of request) above target 2m6s Normal Killing pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Stopping container payload-pre-processing 2m6s Normal Killing pod/payload-processing-e2e-worker-w0-4afe0e-6f5f884484-nm8b6 Stopping container payload-processing 2m6s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj 2m6s Normal ScalingReplicaSet deployment/e2e-worker-w4-64c20d-openshift-default Scaled up replica set e2e-worker-w4-64c20d-openshift-default-d97b6c887 from 6 to 8 2m6s Normal SuccessfulCreate replicaset/e2e-worker-w4-64c20d-openshift-default-d97b6c887 Created pod: e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 2m5s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Container created 2m5s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Container started 2m5s Normal ScalingReplicaSet deployment/istiod-openshift-gateway Scaled up replica set istiod-openshift-gateway-7ddd9cfdc8 from 1 to 2 2m5s Normal SuccessfulRescale horizontalpodautoscaler/istiod-openshift-gateway New size: 2; reason: cpu resource utilization (percentage of request) above target 2m5s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Add eth0 [10.134.0.77/23] from ovn-kubernetes 2m5s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m5s Normal AddedInterface pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Add eth0 [10.133.0.65/23] from ovn-kubernetes 2m5s Normal SuccessfulCreate replicaset/istiod-openshift-gateway-7ddd9cfdc8 Created pod: istiod-openshift-gateway-7ddd9cfdc8-5ck24 2m5s Normal Started pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Container started 2m5s Normal Created pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Container created 2m5s Normal Pulled pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" already present on machine and can be accessed by the pod 2m3s Normal AddedInterface pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Add eth0 [10.132.0.85/23] from ovn-kubernetes 2m3s Normal Pulling pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" 2m2s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w5-56a44f-6cc95bbdbc-sbmmp Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 119s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Startup probe failed: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused 119s Normal Pulled pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:d5adf3101d4e14c710cd05afb3fa53498e3c5fb31a120f1bb007fba2e3b376f8" in 3.635s (3.635s including waiting). Image size: 185476477 bytes. 119s Normal Created pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Container created 119s Normal Started pod/istiod-openshift-gateway-7ddd9cfdc8-5ck24 Container started 119s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Startup probe error: Get "http://10.133.0.65:15021/healthz/ready": dial tcp 10.133.0.65:15021: connect: connection refused... 118s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Startup probe failed: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused 118s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Startup probe error: Get "http://10.134.0.77:15021/healthz/ready": dial tcp 10.134.0.77:15021: connect: connection refused... 116s Warning Unhealthy pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Startup probe failed: Get "http://10.133.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 116s Warning ProbeError pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Startup probe error: Get "http://10.133.0.65:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-kttqn Stopping container istio-proxy 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-hvn5c Stopping container istio-proxy 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-xtxjs Stopping container istio-proxy 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Stopping container istio-proxy 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-dc2k8 Stopping container istio-proxy 111s Normal Killing pod/e2e-worker-w5-56a44f-openshift-default-584894595f-fgc2j Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-hp7nb Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-jfwfj Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-6h442 Stopping container istio-proxy 109s Warning Unhealthy pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Readiness probe failed: HTTP probe failed with statuscode: 503 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-bbdqf Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-w49zv Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-klv9s Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-fxfhn Stopping container istio-proxy 109s Normal Killing pod/e2e-worker-w4-64c20d-openshift-default-d97b6c887-k5lnv Stopping container istio-proxy 109s Warning ProbeError pod/e2e-worker-w5-56a44f-openshift-default-584894595f-f866m Readiness probe error: HTTP probe failed with statuscode: 503... 103s Warning Unhealthy pod/payload-pre-processing-e2e-worker-w0-4afe0e-5979ffbdff-c88st Readiness probe failed: service unhealthy (responded with "NOT_SERVING") 101s Normal Killing pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-j4cjd Stopping container istio-proxy 101s Normal Killing pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-cwvw2 Stopping container istio-proxy 101s Normal Killing pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-zjwdd Stopping container istio-proxy 101s Normal Killing pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Stopping container istio-proxy 96s Warning Unhealthy pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Readiness probe failed: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused 96s Warning ProbeError pod/e2e-worker-w0-4afe0e-openshift-default-5f76c4b57f-6tdsx Readiness probe error: Get "http://10.134.0.38:15021/healthz/ready": dial tcp 10.134.0.38:15021: connect: connection refused...