Time Namespace Component RelatedObject Reason Message

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-dfrnm to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589 to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz to ip-10-0-132-91.ec2.internal

kuadrant-system

limitador-operator-controller-manager-6f9f468797-tghfc

Scheduled

Successfully assigned kuadrant-system/limitador-operator-controller-manager-6f9f468797-tghfc to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq to ip-10-0-132-91.ec2.internal

kuadrant-system

authorino-b6c89bb85-xv7jv

Scheduled

Successfully assigned kuadrant-system/authorino-b6c89bb85-xv7jv to ip-10-0-132-91.ec2.internal

kuadrant-system

authorino-operator-6d85f6564-hfksn

Scheduled

Successfully assigned kuadrant-system/authorino-operator-6d85f6564-hfksn to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9 to ip-10-0-135-91.ec2.internal

kuadrant-system

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

Scheduled

Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg to ip-10-0-132-91.ec2.internal

openshift-ingress

router-default-5d998f6f9d-t8frx

Scheduled

Successfully assigned openshift-ingress/router-default-5d998f6f9d-t8frx to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn to ip-10-0-143-17.ec2.internal

kuadrant-system

dns-operator-controller-manager-65b49595d7-zcjhd

Scheduled

Successfully assigned kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2 to ip-10-0-132-91.ec2.internal

kuadrant-system

limitador-limitador-ff68ff5cb-zkphb

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-ff68ff5cb-zkphb to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w to ip-10-0-132-91.ec2.internal

kuadrant-system

limitador-limitador-69574b596d-x7mgm

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-69574b596d-x7mgm to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7 to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-f9cdc8f48-tn5vx

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-f9cdc8f48-92dw2

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2 to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Scheduled

Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8 to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm to ip-10-0-132-91.ec2.internal

kuadrant-system

kuadrant-console-plugin-85fd4b8b9d-c9n2w

Scheduled

Successfully assigned kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-c9n2w to ip-10-0-135-91.ec2.internal

openshift-ingress

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Scheduled

Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8 to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

Scheduled

Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6 to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

router-gateway-1-openshift-default-799f46c59b-sf6qk

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-sf6qk to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-76b768cb56-46p5r

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5 to ip-10-0-132-91.ec2.internal

kuadrant-system

authorino-7c77957575-pqfjx

Scheduled

Successfully assigned kuadrant-system/authorino-7c77957575-pqfjx to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 to ip-10-0-132-91.ec2.internal

openshift-ingress

istiod-openshift-gateway-94bb8fbfd-6h99s

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6h99s to ip-10-0-143-17.ec2.internal

kuadrant-system

authorino-8df697c9f-p4wg4

Scheduled

Successfully assigned kuadrant-system/authorino-8df697c9f-p4wg4 to ip-10-0-132-91.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77 to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 to ip-10-0-143-17.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx to ip-10-0-135-91.ec2.internal

kserve-ci-e2e-test

router-gateway-2-openshift-default-54c789bdc6-66nfj

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-66nfj to ip-10-0-143-17.ec2.internal

openshift-ingress

istiod-openshift-gateway-7574459656-kjkln

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7574459656-kjkln to ip-10-0-143-17.ec2.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-5d998f6f9d from 0 to 1
(x8)

openshift-ingress

replicaset-controller

router-default-5d998f6f9d

FailedCreate

Error creating: pods "router-default-5d998f6f9d-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x16)

openshift-ingress

replicaset-controller

router-default-5d998f6f9d

FailedCreate

Error creating: pods "router-default-5d998f6f9d-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts

openshift-ingress

replicaset-controller

router-default-5d998f6f9d

SuccessfulCreate

Created pod: router-default-5d998f6f9d-t8frx
(x6)

openshift-ingress

kubelet

router-default-5d998f6f9d-t8frx

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
(x6)

openshift-ingress

kubelet

router-default-5d998f6f9d-t8frx

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt

openshift-ingress

multus

router-default-5d998f6f9d-t8frx

AddedInterface

Add eth0 [10.133.0.9/23] from ovn-kubernetes

openshift-ingress

kubelet

router-default-5d998f6f9d-t8frx

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0595b1d6de8e08ddb61ce581075357331193db6206d341c15cf213514ef0f693" already present on machine

openshift-ingress

kubelet

router-default-5d998f6f9d-t8frx

Created

Created container: router

openshift-ingress

kubelet

router-default-5d998f6f9d-t8frx

Started

Started container router

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7574459656 from 0 to 1

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7574459656

SuccessfulCreate

Created pod: istiod-openshift-gateway-7574459656-kjkln

openshift-ingress

multus

istiod-openshift-gateway-7574459656-kjkln

AddedInterface

Add eth0 [10.133.0.33/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9"

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9" in 2.973s (2.973s including waiting). Image size: 181034515 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

deployment-controller

openshift-ai-inference-openshift-default

ScalingReplicaSet

Scaled up replica set openshift-ai-inference-openshift-default-6c48cf8bcc from 0 to 1

openshift-ingress

gateway_labeler_controller

openshift-ai-inference

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

replicaset-controller

openshift-ai-inference-openshift-default-6c48cf8bcc

SuccessfulCreate

Created pod: openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

openshift-ingress

multus

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

AddedInterface

Add eth0 [10.133.0.34/23] from ovn-kubernetes

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8"

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuredLoadBalancer

Ensured load balancer

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.592s (2.592s including waiting). Image size: 181251556 bytes.

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

InstallWaiting

installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

kubelet

limitador-operator-controller-manager-6f9f468797-tghfc

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e"

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

RequirementsUnknown

requirements not yet checked

kuadrant-system

deployment-controller

limitador-operator-controller-manager

ScalingReplicaSet

Scaled up replica set limitador-operator-controller-manager-6f9f468797 from 0 to 1

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

replicaset-controller

limitador-operator-controller-manager-6f9f468797

SuccessfulCreate

Created pod: limitador-operator-controller-manager-6f9f468797-tghfc

kuadrant-system

multus

limitador-operator-controller-manager-6f9f468797-tghfc

AddedInterface

Add eth0 [10.132.0.25/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

AllRequirementsMet

all requirements found, attempting install
(x2)

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

deployment-controller

authorino-operator

ScalingReplicaSet

Scaled up replica set authorino-operator-6d85f6564 from 0 to 1

kuadrant-system

multus

authorino-operator-6d85f6564-hfksn

AddedInterface

Add eth0 [10.133.0.40/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-operator-6d85f6564-hfksn

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687"

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

InstallWaiting

installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

replicaset-controller

authorino-operator-6d85f6564

SuccessfulCreate

Created pod: authorino-operator-6d85f6564-hfksn

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

RequirementsUnknown

requirements not yet checked

kuadrant-system

replicaset-controller

dns-operator-controller-manager-65b49595d7

SuccessfulCreate

Created pod: dns-operator-controller-manager-65b49595d7-zcjhd

kuadrant-system

deployment-controller

dns-operator-controller-manager

ScalingReplicaSet

Scaled up replica set dns-operator-controller-manager-65b49595d7 from 0 to 1

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

InstallWaiting

installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

kubelet

dns-operator-controller-manager-65b49595d7-zcjhd

Pulling

Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5"

kuadrant-system

multus

dns-operator-controller-manager-65b49595d7-zcjhd

AddedInterface

Add eth0 [10.134.0.19/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

kubelet

authorino-operator-6d85f6564-hfksn

Started

Started container manager

kuadrant-system

kubelet

authorino-operator-6d85f6564-hfksn

Created

Created container: manager

kuadrant-system

kubelet

authorino-operator-6d85f6564-hfksn

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:b2cc2c0f0729dad98f970830816e300c0b961ba42491f362295512d886384687" in 2.523s (2.523s including waiting). Image size: 168639631 bytes.

kuadrant-system

authorino-operator-6d85f6564-hfksn_b20b6f24-06b8-4190-8d12-c55623772eac

aac3a15d.authorino.kuadrant.io

LeaderElection

authorino-operator-6d85f6564-hfksn_b20b6f24-06b8-4190-8d12-c55623772eac became leader

kuadrant-system

kubelet

dns-operator-controller-manager-65b49595d7-zcjhd

Created

Created container: manager

kuadrant-system

kubelet

dns-operator-controller-manager-65b49595d7-zcjhd

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:ba46f54446ecd3ed1a3d877dcbb7359ff8f2d918827f3411a5cb0781915a65d5" in 1.184s (1.184s including waiting). Image size: 190472853 bytes.

kuadrant-system

dns-operator-controller-manager-65b49595d7-zcjhd_32152c80-88c5-4e1c-b4ff-54704bfc6b6a

a3f98d6c.kuadrant.io

LeaderElection

dns-operator-controller-manager-65b49595d7-zcjhd_32152c80-88c5-4e1c-b4ff-54704bfc6b6a became leader

kuadrant-system

kubelet

dns-operator-controller-manager-65b49595d7-zcjhd

Started

Started container manager

kuadrant-system

multus

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

AddedInterface

Add eth0 [10.134.0.20/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

InstallWaiting

installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

deployment-controller

kuadrant-operator-controller-manager

ScalingReplicaSet

Scaled up replica set kuadrant-operator-controller-manager-bb7ddb7d7 from 0 to 1

kuadrant-system

replicaset-controller

kuadrant-operator-controller-manager-bb7ddb7d7

SuccessfulCreate

Created pod: kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

kuadrant-system

kubelet

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3"

kuadrant-system

kubelet

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

Started

Started container manager

kuadrant-system

kubelet

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

Created

Created container: manager

kuadrant-system

kubelet

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:0b77e4a874b620095a0300877aea53e78559a601ce62b194537ed21c2743bbf3" in 3.782s (3.782s including waiting). Image size: 425661757 bytes.

kuadrant-system

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj_63cfd7c8-d30b-4f3c-80cc-38b7be84ec81

f139389e.kuadrant.io

LeaderElection

kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj_63cfd7c8-d30b-4f3c-80cc-38b7be84ec81 became leader

kuadrant-system

deployment-controller

kuadrant-console-plugin

ScalingReplicaSet

Scaled up replica set kuadrant-console-plugin-85fd4b8b9d from 0 to 1

kuadrant-system

multus

kuadrant-console-plugin-85fd4b8b9d-c9n2w

AddedInterface

Add eth0 [10.132.0.26/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-console-plugin-85fd4b8b9d-c9n2w

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a"

kuadrant-system

replicaset-controller

kuadrant-console-plugin-85fd4b8b9d

SuccessfulCreate

Created pod: kuadrant-console-plugin-85fd4b8b9d-c9n2w

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.4.1

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.4.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

kubelet

limitador-operator-controller-manager-6f9f468797-tghfc

Started

Started container manager

kuadrant-system

kubelet

limitador-operator-controller-manager-6f9f468797-tghfc

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:183bad2a6f5872aea38f54ad507f7e005d6facb02b18769e0407c82799f96a3e" in 20.719s (20.719s including waiting). Image size: 178538972 bytes.

kuadrant-system

kubelet

kuadrant-console-plugin-85fd4b8b9d-c9n2w

Started

Started container kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-85fd4b8b9d-c9n2w

Created

Created container: kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-85fd4b8b9d-c9n2w

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:6e69f41d3d70b0afa7691a5d2400065d3bcdec2ea82830cf2992b8e69ba06e2a" in 5.123s (5.123s including waiting). Image size: 342502139 bytes.

kuadrant-system

limitador-operator-controller-manager-6f9f468797-tghfc_6712bf34-bf56-45c1-b446-dfff9a612bdc

3745a16e.kuadrant.io

LeaderElection

limitador-operator-controller-manager-6f9f468797-tghfc_6712bf34-bf56-45c1-b446-dfff9a612bdc became leader

kuadrant-system

kubelet

limitador-operator-controller-manager-6f9f468797-tghfc

Created

Created container: manager

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.4.1

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.4.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

replicaset-controller

limitador-limitador-ff68ff5cb

SuccessfulCreate

Created pod: limitador-limitador-ff68ff5cb-zkphb

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-ff68ff5cb from 0 to 1

kuadrant-system

kubelet

limitador-limitador-69574b596d-x7mgm

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"

kuadrant-system

multus

limitador-limitador-69574b596d-x7mgm

AddedInterface

Add eth0 [10.133.0.41/23] from ovn-kubernetes

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-b6c89bb85 from 0 to 1

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-8df697c9f from 0 to 1

kuadrant-system

replicaset-controller

authorino-b6c89bb85

SuccessfulCreate

Created pod: authorino-b6c89bb85-xv7jv

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-69574b596d from 0 to 1

kuadrant-system

replicaset-controller

limitador-limitador-69574b596d

SuccessfulCreate

Created pod: limitador-limitador-69574b596d-x7mgm

kuadrant-system

multus

limitador-limitador-ff68ff5cb-zkphb

AddedInterface

Add eth0 [10.132.0.27/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-limitador-ff68ff5cb-zkphb

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"

kuadrant-system

replicaset-controller

authorino-8df697c9f

SuccessfulCreate

Created pod: authorino-8df697c9f-p4wg4

kuadrant-system

kubelet

authorino-8df697c9f-p4wg4

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"

kuadrant-system

multus

authorino-b6c89bb85-xv7jv

AddedInterface

Add eth0 [10.134.0.21/23] from ovn-kubernetes

kuadrant-system

multus

authorino-8df697c9f-p4wg4

AddedInterface

Add eth0 [10.134.0.22/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-b6c89bb85-xv7jv

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"

kuadrant-system

kubelet

authorino-8df697c9f-p4wg4

Created

Created container: authorino

kuadrant-system

kubelet

authorino-8df697c9f-p4wg4

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 2.519s (2.519s including waiting). Image size: 202706732 bytes.

kuadrant-system

kubelet

authorino-b6c89bb85-xv7jv

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 2.625s (2.625s including waiting). Image size: 202706732 bytes.

kuadrant-system

kubelet

authorino-b6c89bb85-xv7jv

Started

Started container authorino

kuadrant-system

kubelet

authorino-b6c89bb85-xv7jv

Created

Created container: authorino

kuadrant-system

kubelet

authorino-8df697c9f-p4wg4

Started

Started container authorino

kuadrant-system

replicaset-controller

authorino-b6c89bb85

SuccessfulDelete

Deleted pod: authorino-b6c89bb85-xv7jv

kuadrant-system

kubelet

limitador-limitador-ff68ff5cb-zkphb

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 3.866s (3.866s including waiting). Image size: 250499765 bytes.

kuadrant-system

kubelet

limitador-limitador-69574b596d-x7mgm

Created

Created container: limitador

kuadrant-system

kubelet

limitador-limitador-69574b596d-x7mgm

Started

Started container limitador

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-b6c89bb85 from 1 to 0

kuadrant-system

kubelet

limitador-limitador-69574b596d-x7mgm

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474" in 3.554s (3.554s including waiting). Image size: 250499765 bytes.

kuadrant-system

kubelet

limitador-limitador-ff68ff5cb-zkphb

Started

Started container limitador

kuadrant-system

kubelet

limitador-limitador-ff68ff5cb-zkphb

Created

Created container: limitador

kuadrant-system

kubelet

authorino-b6c89bb85-xv7jv

Killing

Stopping container authorino

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled down replica set limitador-limitador-ff68ff5cb from 1 to 0

kuadrant-system

kubelet

limitador-limitador-ff68ff5cb-zkphb

Killing

Stopping container limitador

kuadrant-system

replicaset-controller

limitador-limitador-ff68ff5cb

SuccessfulDelete

Deleted pod: limitador-limitador-ff68ff5cb-zkphb

kuadrant-system

multus

authorino-7c77957575-pqfjx

AddedInterface

Add eth0 [10.134.0.23/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-7c77957575-pqfjx

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a"

kuadrant-system

replicaset-controller

authorino-7c77957575

SuccessfulCreate

Created pod: authorino-7c77957575-pqfjx

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-7c77957575 from 0 to 1

kuadrant-system

kubelet

authorino-7c77957575-pqfjx

Created

Created container: authorino

kuadrant-system

kubelet

authorino-8df697c9f-p4wg4

Killing

Stopping container authorino

kuadrant-system

kubelet

authorino-7c77957575-pqfjx

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:eaadd50c0e4938926e50283a5f3e3b5f1dd24d820a3a2c7434153080180cc95a" in 496ms (496ms including waiting). Image size: 202706732 bytes.

kuadrant-system

kubelet

authorino-7c77957575-pqfjx

Started

Started container authorino

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-8df697c9f from 1 to 0

kuadrant-system

replicaset-controller

authorino-8df697c9f

SuccessfulDelete

Deleted pod: authorino-8df697c9f-p4wg4

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-94bb8fbfd from 0 to 1

openshift-ingress

replicaset-controller

istiod-openshift-gateway-94bb8fbfd

SuccessfulCreate

Created pod: istiod-openshift-gateway-94bb8fbfd-6h99s

openshift-ingress

multus

istiod-openshift-gateway-94bb8fbfd-6h99s

AddedInterface

Add eth0 [10.133.0.42/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-94bb8fbfd-6h99s

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-94bb8fbfd-6h99s

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-94bb8fbfd-6h99s

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:0850242436e88f7d82f0f2126de064c7e0f09844f31d8ff0f53dc8d3908075d9" already present on machine

openshift-ingress

kubelet

istiod-openshift-gateway-7574459656-kjkln

Killing

Stopping container discovery

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7574459656

SuccessfulDelete

Deleted pod: istiod-openshift-gateway-7574459656-kjkln

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled down replica set istiod-openshift-gateway-7574459656 from 1 to 0
(x4)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

deployment-controller

router-gateway-1-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

multus

router-gateway-1-openshift-default-799f46c59b-sf6qk

AddedInterface

Add eth0 [10.133.0.44/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

router-gateway-1-openshift-default-799f46c59b

SuccessfulCreate

Created pod: router-gateway-1-openshift-default-799f46c59b-sf6qk

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuredLoadBalancer

Ensured load balancer
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-inline-config-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-f96d8754d

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-f96d8754d-2tjs2

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-779f9bd675 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-f96d8754d from 0 to 1

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-router-scheduler-779f9bd675

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

AddedInterface

Add eth0 [10.134.0.25/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812"

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

AddedInterface

Add eth0 [10.132.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" in 3.324s (3.324s including waiting). Image size: 302769600 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" in 2.921s (2.921s including waiting). Image size: 302769600 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.001s (1.001s including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Created

Created container: main

kserve-ci-e2e-test

multus

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

AddedInterface

Add eth0 [10.133.0.45/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

gw-section-name-router-with-gat-2f0a622e-kserve

ScalingReplicaSet

Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc

kserve-ci-e2e-test

replicaset-controller

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559

SuccessfulCreate

Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812"

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.118s (1.118s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" in 3.954s (3.954s including waiting). Image size: 302769600 bytes.

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Created

Created container: storage-initializer
(x2)

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Started

Started container istio-proxy

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
(x2)

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Created

Created container: istio-proxy
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-799f46c59b-sf6qk

Created

Created container: istio-proxy
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-799f46c59b-sf6qk

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-799f46c59b-sf6qk

Started

Started container istio-proxy

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.566s (1.566s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Created

Created container: main
(x6)

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

Unhealthy

Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused
(x6)

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6c48cf8bcc-cl5g8

ProbeError

Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused body:

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Started

Started container main
(x5)

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-799f46c59b-sf6qk

Unhealthy

Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-798cdbd559pb2cr

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route

kserve-ci-e2e-test

replicaset-controller

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fb

SuccessfulCreate

Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

gw-section-name-router-with-gat-f1d92d0f-kserve

ScalingReplicaSet

Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fb from 0 to 1

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

AddedInterface

Add eth0 [10.133.0.46/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-7d7fd586fbvqjx6

Killing

Stopping container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.458s (30.458s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Created

Created container: tokenizer
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-6fa1027a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44 from 0 to 1

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

AddedInterface

Add eth0 [10.133.0.47/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-779f9swl6k

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-f96d8754d-2tjs2

Killing

Stopping container main
(x10)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-configmap-ref-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-6555bfb5c6

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-6555bfb5c6 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

AddedInterface

Add eth0 [10.132.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

AddedInterface

Add eth0 [10.134.0.26/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-85bdf5c8b9 from 0 to 1

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-router-scheduler-85bdf5c8b9

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Started

Started container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-85bdfb7pcl

Killing

Stopping container main
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-ha-replicas-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-d9c8d8748 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-d9c8d8748

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

AddedInterface

Add eth0 [10.134.0.28/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

AddedInterface

Add eth0 [10.132.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

AddedInterface

Add eth0 [10.134.0.27/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf66f from 0 to 2

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Created

Created container: storage-initializer

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf66f

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf66f

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 921ms (921ms including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.027s (30.027s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Killing

Stopping container main
(x3)

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bf7fz77

Killing

Stopping container main
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

precise-prefix-cache-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-6657b58549

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-6657b58549-dfrnm

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-6657b58549

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-6657b58549-hk5t7

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-6657b58549 from 0 to 2

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-router-scheduler-594c699cf

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Started

Started container storage-initializer

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-594c699cf from 0 to 1

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

AddedInterface

Add eth0 [10.132.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-6657b58549-hk5t7

AddedInterface

Add eth0 [10.134.0.29/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-6657b58549-dfrnm

AddedInterface

Add eth0 [10.132.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.189s (1.189s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Updated

Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc
(x3)

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.134.0.28:9003" within 1s: context deadline exceeded
(x2)

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Unhealthy

Liveness probe failed: timeout: failed to connect service "10.132.0.37:9003" within 1s: context deadline exceeded

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-dfrnm

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-v1a1-to-v1a2

Error

Reconciliation failed: failed to combine base-configurations: failed to get LLMInferenceServiceConfig "conv-test-v1a1-to-v1a2-config" from namespaces ["kserve-ci-e2e-test", "kserve"]: LLMInferenceServiceConfig.serving.kserve.io "conv-test-v1a1-to-v1a2-config" not found

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-6657b58549-hk5t7

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-594c699c7q589

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-round-trip

Error

Reconciliation failed: failed to combine base-configurations: failed to get LLMInferenceServiceConfig "conv-test-round-trip-config" from namespaces ["kserve-ci-e2e-test", "kserve"]: LLMInferenceServiceConfig.serving.kserve.io "conv-test-round-trip-config" not found

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-criticality

Error

Reconciliation failed: failed to combine base-configurations: failed to get LLMInferenceServiceConfig "conv-test-criticality-config" from namespaces ["kserve-ci-e2e-test", "kserve"]: LLMInferenceServiceConfig.serving.kserve.io "conv-test-criticality-config" not found

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-f9cdc8f48

SuccessfulCreate

Created pod: stop-feature-test-kserve-f9cdc8f48-tn5vx

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-f9cdc8f48 from 0 to 1

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-5764dd495b from 0 to 1

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-5764dd495b

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

AddedInterface

Add eth0 [10.132.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

stop-feature-test-kserve-f9cdc8f48-tn5vx

AddedInterface

Add eth0 [10.134.0.30/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Updated

Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.222s (27.222s including waiting). Image size: 3531177328 bytes.

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Started

Started container main

kserve-ci-e2e-test

GatewayAuthBootstrap

router-gateway-1

ReconcileError

Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-1-metrics" not found

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-66b8d9fc44xrxgt

Killing

Stopping container main
(x9)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Unhealthy

Startup probe failed: Get "https://10.134.0.30:8000/health": dial tcp 10.134.0.30:8000: connect: connection refused
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-55f7ae4a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route" not found

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

AddedInterface

Add eth0 [10.134.0.31/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa

kserve-ci-e2e-test

deployment-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-689676db46 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-689676db46

SuccessfulCreate

Created pod: llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-workload-svc

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796 from 0 to 1

kserve-ci-e2e-test

multus

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

AddedInterface

Add eth0 [10.134.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-qx6xg

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-tn5vx

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1alpha2.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool
(x21)

kserve-ci-e2e-test

OpenDataHubModelController

stop-feature-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-f9cdc8f48

SuccessfulCreate

Created pod: stop-feature-test-kserve-f9cdc8f48-92dw2

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-f9cdc8f48 from 0 to 1

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-5764dd495b from 0 to 1

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-5764dd495b

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

AddedInterface

Add eth0 [10.132.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

stop-feature-test-kserve-f9cdc8f48-92dw2

AddedInterface

Add eth0 [10.134.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
(x11)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Unhealthy

Startup probe failed: Get "https://10.134.0.31:8000/health": dial tcp 10.134.0.31:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq

Killing

Stopping container main
(x10)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-fb9546c6d from 0 to 1

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

AddedInterface

Add eth0 [10.134.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-fb9546c6d

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-fb9546c6d-zzrhg

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-router-scheduler-6bf8f557f8

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

AddedInterface

Add eth0 [10.132.0.40/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-6bf8f557f8 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
(x11)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Unhealthy

Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-f9cdc8f48-92dw2

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-5764dd495b-85p5b

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.132.0.39:9003" within 1s: context deadline exceeded
(x9)

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Unhealthy

Startup probe failed: Get "https://10.134.0.34:8000/health": dial tcp 10.134.0.34:8000: connect: connection refused

kserve-ci-e2e-test

replicaset-controller

router-gateway-2-openshift-default-54c789bdc6

SuccessfulCreate

Created pod: router-gateway-2-openshift-default-54c789bdc6-66nfj

kserve-ci-e2e-test

multus

router-gateway-2-openshift-default-54c789bdc6-66nfj

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

router-gateway-2-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-2

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-2

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuredLoadBalancer

Ensured load balancer
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-54c789bdc6-66nfj

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-54c789bdc6-66nfj

Started

Started container istio-proxy
(x2)

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-54c789bdc6-66nfj

Created

Created container: istio-proxy
(x5)

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-54c789bdc6-66nfj

Unhealthy

Startup probe failed: Get "http://10.133.0.49:15021/healthz/ready": dial tcp 10.133.0.49:15021: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-prefill-765c869bbf

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-prefill-765c869bbf from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-6fd47695bb

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-6fd47695bb from 0 to 1

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87c from 0 to 1

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

AddedInterface

Add eth0 [10.134.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

AddedInterface

Add eth0 [10.132.0.41/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87c

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

AddedInterface

Add eth0 [10.134.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
(x8)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 790ms (790ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Started

Started container main
(x4)

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-1

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-fb9546c6d-zzrhg

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-6bf8f557g698m

Killing

Stopping container tokenizer

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-76b768cb56

SuccessfulCreate

Created pod: router-with-refs-test-kserve-76b768cb56-46p5r

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-76b768cb56 from 0 to 1

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d99ffd6c4 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-76b768cb56-46p5r

AddedInterface

Add eth0 [10.134.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

AddedInterface

Add eth0 [10.132.0.42/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4

SuccessfulCreate

Created pod: router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role
(x4)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Started

Started container main
(x2)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
(x16)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Unhealthy

Startup probe failed: Get "https://10.134.0.36:8000/health": dial tcp 10.134.0.36:8000: connect: connection refused
(x16)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Unhealthy

Startup probe failed: Get "https://10.134.0.35:8001/health": dial tcp 10.134.0.35:8001: connect: connection refused
(x13)

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Unhealthy

Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cb9cd87cspl4v

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-6fd47695bb-j7qrz

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb
(x9)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-02ac88e7

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

AddedInterface

Add eth0 [10.134.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Started

Started container main
(x2)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-7d99ffd6c4-ffzvx

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-76b768cb56-46p5r

Killing

Stopping container main
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-05aa9bba

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-rb

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

deployment-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-6b57744674 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

kserve-ci-e2e-test

multus

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

AddedInterface

Add eth0 [10.132.0.43/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

replicaset-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-6b57744674

SuccessfulCreate

Created pod: llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

AddedInterface

Add eth0 [10.134.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bcf675468

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-workload-svc

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

AddedInterface

Add eth0 [10.134.0.40/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Pulled

Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bcf675468 from 0 to 1

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route
(x10)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Unhealthy

Startup probe failed: Get "https://10.134.0.38:8000/health": dial tcp 10.134.0.38:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg

Killing

Stopping container main
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-f312f5ec

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

deployment-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-5dcddf9db from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-scc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-rb

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn successful
(x2)

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Replicas are progressing, with 0 groups ready of total 1 groups

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created leader statefulset llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

CreatingRevision

Creating revision with key 745558bb58 for a newly created LeaderWorkerSet

kserve-ci-e2e-test

replicaset-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-5dcddf9db

SuccessfulCreate

Created pod: llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 successful

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

AddedInterface

Add eth0 [10.134.0.41/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

AddedInterface

Add eth0 [10.132.0.44/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
(x8)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.385s (27.385s including waiting). Image size: 3531177328 bytes.

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: main
(x14)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Unhealthy

Startup probe failed: Get "https://10.134.0.39:8001/health": dial tcp 10.134.0.39:8001: connect: connection refused
(x15)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Unhealthy

Startup probe failed: Get "https://10.134.0.40:8000/health": dial tcp 10.134.0.40:8000: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-sches8ktc

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm

Killing

Stopping container llm-d-routing-sidecar
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-pd-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

AddedInterface

Add eth0 [10.133.0.51/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-d9768b96b

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

AddedInterface

Add eth0 [10.133.0.52/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-prefill-7d465b7985

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7d465b7985 from 0 to 1

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

AddedInterface

Add eth0 [10.132.0.45/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-d9768b96b from 0 to 1

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-c5c7bcb5c from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7bcb5c

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:e7d18b8520466922c73925c5601004b31b3c3ec90f89a0821d13bdeee1848812" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 866ms (866ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Created

Created container: main
(x11)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Unhealthy

Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Updated

Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc
(x2)

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

AllGroupsReady

All replicas are ready, with 1 groups ready of total 1 groups

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schelxsz9

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Killing

Stopping container main
(x13)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Unhealthy

Startup probe failed: Get "https://10.133.0.51:8001/health": dial tcp 10.133.0.51:8001: connect: connection refused
(x13)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Unhealthy

Startup probe failed: Get "https://10.133.0.52:8000/health": dial tcp 10.133.0.52:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-7d465b7985-qljdn

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-c5c7b5hbv8

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-d9768b96b-f4kzh

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb