Time Namespace Component RelatedObject Reason Message

kserve-ci-e2e-test

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7 to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj to ip-10-0-137-126.ec2.internal

openshift-ingress

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-wcfq9 to ip-10-0-137-126.ec2.internal

openshift-ingress

istiod-openshift-gateway-55ff986f96-tvvcc

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-55ff986f96-tvvcc to ip-10-0-143-10.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8657c58775-xld44 to ip-10-0-138-58.ec2.internal

kuadrant-system

kuadrant-console-plugin-6c886788f8-d76cw

Scheduled

Successfully assigned kuadrant-system/kuadrant-console-plugin-6c886788f8-d76cw to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-587498644-znkwq

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-587498644-znkwq to ip-10-0-143-10.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-5698dc8fcf-ng9bf

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-5698dc8fcf-ng9bf to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-5698dc8fcf-zspvz

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-5698dc8fcf-zspvz to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-855cffc847-fzvhl to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-968f7c9f5-dcrlb to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 to ip-10-0-138-58.ec2.internal

kuadrant-system

dns-operator-controller-manager-844548ff4c-6pk8c

Scheduled

Successfully assigned kuadrant-system/dns-operator-controller-manager-844548ff4c-6pk8c to ip-10-0-137-126.ec2.internal

kuadrant-system

authorino-674b59b84c-6kvzj

Scheduled

Successfully assigned kuadrant-system/authorino-674b59b84c-6kvzj to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-5959bdbb55-8fxgx

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5959bdbb55-8fxgx to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr to ip-10-0-138-58.ec2.internal

kuadrant-system

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

Scheduled

Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-jn6hg to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

router-gateway-1-openshift-default-6c59fbf55c-l24v6

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-l24v6 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj to ip-10-0-138-58.ec2.internal

kuadrant-system

authorino-68bd676465-5ttpp

Scheduled

Successfully assigned kuadrant-system/authorino-68bd676465-5ttpp to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz to ip-10-0-137-126.ec2.internal

openshift-ingress

router-default-5887855b54-djpw5

Scheduled

Successfully assigned openshift-ingress/router-default-5887855b54-djpw5 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27 to ip-10-0-137-126.ec2.internal

kuadrant-system

limitador-limitador-64c8f475fb-5ndg6

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-64c8f475fb-5ndg6 to ip-10-0-143-10.ec2.internal

kuadrant-system

limitador-limitador-67566c68b4-lbws6

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-67566c68b4-lbws6 to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l to ip-10-0-137-126.ec2.internal

kuadrant-system

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

Scheduled

Successfully assigned kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-cwpbt to ip-10-0-143-10.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22 to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 to ip-10-0-137-126.ec2.internal

kuadrant-system

authorino-operator-7587b89b76-qrvbs

Scheduled

Successfully assigned kuadrant-system/authorino-operator-7587b89b76-qrvbs to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz to ip-10-0-137-126.ec2.internal

kuadrant-system

authorino-79cbc94b89-64sf6

Scheduled

Successfully assigned kuadrant-system/authorino-79cbc94b89-64sf6 to ip-10-0-138-58.ec2.internal

openshift-ingress

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

Scheduled

Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-pm5bn to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

conv-test-round-trip-kserve-fb84c864-t4g46

Scheduled

Successfully assigned kserve-ci-e2e-test/conv-test-round-trip-kserve-fb84c864-t4g46 to ip-10-0-137-126.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

router-gateway-2-openshift-default-6866b85949-lflmt

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lflmt to ip-10-0-143-10.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7db845475b-cn8h6 to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x to ip-10-0-138-58.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-7db845475b-rngn7

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7db845475b-rngn7 to ip-10-0-137-126.ec2.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-5887855b54 from 0 to 1

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x14)

openshift-ingress

replicaset-controller

router-default-5887855b54

FailedCreate

Error creating: pods "router-default-5887855b54-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes
(x15)

openshift-ingress

replicaset-controller

router-default-5887855b54

FailedCreate

Error creating: pods "router-default-5887855b54-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

openshift-ingress

replicaset-controller

router-default-5887855b54

SuccessfulCreate

Created pod: router-default-5887855b54-djpw5
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts
(x6)

openshift-ingress

kubelet

router-default-5887855b54-djpw5

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
(x6)

openshift-ingress

kubelet

router-default-5887855b54-djpw5

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt

openshift-ingress

kubelet

router-default-5887855b54-djpw5

Started

Started container router

openshift-ingress

multus

router-default-5887855b54-djpw5

AddedInterface

Add eth0 [10.132.0.14/23] from ovn-kubernetes

openshift-ingress

kubelet

router-default-5887855b54-djpw5

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e4763905898d3870f64ebc9721d8d43ae2973f4ba295d48f84e36e6f72d013" already present on machine

openshift-ingress

kubelet

router-default-5887855b54-djpw5

Created

Created container: router

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7cd77c7ffd from 0 to 1

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7cd77c7ffd

SuccessfulCreate

Created pod: istiod-openshift-gateway-7cd77c7ffd-wcfq9

openshift-ingress

multus

istiod-openshift-gateway-7cd77c7ffd-wcfq9

AddedInterface

Add eth0 [10.133.0.31/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8"

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" in 2.378s (2.378s including waiting). Image size: 179322965 bytes.

openshift-ingress

multus

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

AddedInterface

Add eth0 [10.133.0.32/23] from ovn-kubernetes

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

gateway_labeler_controller

openshift-ai-inference

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference

openshift-ingress

replicaset-controller

openshift-ai-inference-openshift-default-7c5447bb76

SuccessfulCreate

Created pod: openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"

openshift-ingress

deployment-controller

openshift-ai-inference-openshift-default

ScalingReplicaSet

Scaled up replica set openshift-ai-inference-openshift-default-7c5447bb76 from 0 to 1

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

Started

Started container istio-proxy

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

Created

Created container: istio-proxy

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-7c5447bb76-pm5bn

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 2.734s (2.734s including waiting). Image size: 163191192 bytes.

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuredLoadBalancer

Ensured load balancer

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

deployment-controller

dns-operator-controller-manager

ScalingReplicaSet

Scaled up replica set dns-operator-controller-manager-844548ff4c from 0 to 1

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallWaiting

installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

multus

dns-operator-controller-manager-844548ff4c-6pk8c

AddedInterface

Add eth0 [10.133.0.37/23] from ovn-kubernetes

kuadrant-system

replicaset-controller

dns-operator-controller-manager-844548ff4c

SuccessfulCreate

Created pod: dns-operator-controller-manager-844548ff4c-6pk8c

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6pk8c

Pulling

Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf"

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6pk8c

Created

Created container: manager

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6pk8c

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf" in 2.395s (2.395s including waiting). Image size: 186790172 bytes.

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallWaiting

installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6pk8c

Started

Started container manager

kuadrant-system

replicaset-controller

authorino-operator-7587b89b76

SuccessfulCreate

Created pod: authorino-operator-7587b89b76-qrvbs

kuadrant-system

deployment-controller

authorino-operator

ScalingReplicaSet

Scaled up replica set authorino-operator-7587b89b76 from 0 to 1

kuadrant-system

dns-operator-controller-manager-844548ff4c-6pk8c_e688a99f-fcd1-4e30-b267-d07cfc8c2936

a3f98d6c.kuadrant.io

LeaderElection

dns-operator-controller-manager-844548ff4c-6pk8c_e688a99f-fcd1-4e30-b267-d07cfc8c2936 became leader

kuadrant-system

kubelet

authorino-operator-7587b89b76-qrvbs

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733"

kuadrant-system

multus

authorino-operator-7587b89b76-qrvbs

AddedInterface

Add eth0 [10.133.0.38/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-operator-7587b89b76-qrvbs

Started

Started container manager

kuadrant-system

kubelet

authorino-operator-7587b89b76-qrvbs

Created

Created container: manager

kuadrant-system

kubelet

authorino-operator-7587b89b76-qrvbs

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733" in 1.511s (1.511s including waiting). Image size: 161991975 bytes.

kuadrant-system

authorino-operator-7587b89b76-qrvbs_4abe38fd-efb0-43d7-b3ec-a29e6acc8698

aac3a15d.authorino.kuadrant.io

LeaderElection

authorino-operator-7587b89b76-qrvbs_4abe38fd-efb0-43d7-b3ec-a29e6acc8698 became leader
(x2)

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallWaiting

installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

deployment-controller

limitador-operator-controller-manager

ScalingReplicaSet

Scaled up replica set limitador-operator-controller-manager-c7fb4c8d5 from 0 to 1

kuadrant-system

replicaset-controller

limitador-operator-controller-manager-c7fb4c8d5

SuccessfulCreate

Created pod: limitador-operator-controller-manager-c7fb4c8d5-cwpbt

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

RequirementsUnknown

requirements not yet checked

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66"

kuadrant-system

multus

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

AddedInterface

Add eth0 [10.134.0.24/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

Created

Created container: manager

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

Started

Started container manager

kuadrant-system

limitador-operator-controller-manager-c7fb4c8d5-cwpbt_bcc7a26a-0ac6-42e2-86e5-b895c755ad92

3745a16e.kuadrant.io

LeaderElection

limitador-operator-controller-manager-c7fb4c8d5-cwpbt_bcc7a26a-0ac6-42e2-86e5-b895c755ad92 became leader

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-cwpbt

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66" in 2.439s (2.439s including waiting). Image size: 175506011 bytes.

kuadrant-system

deployment-controller

kuadrant-operator-controller-manager

ScalingReplicaSet

Scaled up replica set kuadrant-operator-controller-manager-6ddf9554fc from 0 to 1

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallWaiting

installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

replicaset-controller

kuadrant-operator-controller-manager-6ddf9554fc

SuccessfulCreate

Created pod: kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

kuadrant-system

multus

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

AddedInterface

Add eth0 [10.133.0.39/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8"

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8" in 4.353s (4.353s including waiting). Image size: 411782238 bytes.

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

Started

Started container manager

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg

Created

Created container: manager

kuadrant-system

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg_ff576c92-209c-4873-b770-848fa422ce03

f139389e.kuadrant.io

LeaderElection

kuadrant-operator-controller-manager-6ddf9554fc-jn6hg_ff576c92-209c-4873-b770-848fa422ce03 became leader

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

deployment-controller

kuadrant-console-plugin

ScalingReplicaSet

Scaled up replica set kuadrant-console-plugin-6c886788f8 from 0 to 1

kuadrant-system

replicaset-controller

kuadrant-console-plugin-6c886788f8

SuccessfulCreate

Created pod: kuadrant-console-plugin-6c886788f8-d76cw

kuadrant-system

multus

kuadrant-console-plugin-6c886788f8-d76cw

AddedInterface

Add eth0 [10.132.0.24/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-d76cw

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57"

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-d76cw

FailedMount

MountVolume.SetUp failed for volume "plugin-serving-cert" : secret "plugin-serving-cert" not found

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallSucceeded

install strategy completed with no errors

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-d76cw

Started

Started container kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-d76cw

Created

Created container: kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-d76cw

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57" in 14.308s (14.308s including waiting). Image size: 339541674 bytes.

kuadrant-system

replicaset-controller

limitador-limitador-64c8f475fb

SuccessfulCreate

Created pod: limitador-limitador-64c8f475fb-5ndg6

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-64c8f475fb from 0 to 1

kuadrant-system

multus

limitador-limitador-64c8f475fb-5ndg6

AddedInterface

Add eth0 [10.134.0.26/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-limitador-67566c68b4-lbws6

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"

kuadrant-system

replicaset-controller

authorino-674b59b84c

SuccessfulCreate

Created pod: authorino-674b59b84c-6kvzj

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-67566c68b4 from 0 to 1

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-5ndg6

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"

kuadrant-system

replicaset-controller

limitador-limitador-67566c68b4

SuccessfulCreate

Created pod: limitador-limitador-67566c68b4-lbws6

kuadrant-system

multus

limitador-limitador-67566c68b4-lbws6

AddedInterface

Add eth0 [10.133.0.40/23] from ovn-kubernetes

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-79cbc94b89 from 0 to 1

kuadrant-system

replicaset-controller

authorino-79cbc94b89

SuccessfulCreate

Created pod: authorino-79cbc94b89-64sf6

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-674b59b84c from 0 to 1

kuadrant-system

kubelet

authorino-79cbc94b89-64sf6

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

multus

authorino-674b59b84c-6kvzj

AddedInterface

Add eth0 [10.132.0.25/23] from ovn-kubernetes

kuadrant-system

multus

authorino-79cbc94b89-64sf6

AddedInterface

Add eth0 [10.132.0.26/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-674b59b84c-6kvzj

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

kubelet

authorino-79cbc94b89-64sf6

Created

Created container: authorino

kuadrant-system

kubelet

authorino-79cbc94b89-64sf6

Started

Started container authorino

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-5ndg6

Created

Created container: limitador

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-5ndg6

Started

Started container limitador

kuadrant-system

kubelet

authorino-674b59b84c-6kvzj

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 2.592s (2.592s including waiting). Image size: 199256829 bytes.

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-5ndg6

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 3.633s (3.633s including waiting). Image size: 249943843 bytes.

kuadrant-system

kubelet

authorino-79cbc94b89-64sf6

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 2.441s (2.441s including waiting). Image size: 199256829 bytes.

kuadrant-system

kubelet

authorino-674b59b84c-6kvzj

Started

Started container authorino

kuadrant-system

kubelet

authorino-674b59b84c-6kvzj

Created

Created container: authorino

kuadrant-system

replicaset-controller

authorino-674b59b84c

SuccessfulDelete

Deleted pod: authorino-674b59b84c-6kvzj

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-674b59b84c from 1 to 0

kuadrant-system

kubelet

authorino-674b59b84c-6kvzj

Killing

Stopping container authorino

kuadrant-system

kubelet

limitador-limitador-67566c68b4-lbws6

Started

Started container limitador

kuadrant-system

kubelet

limitador-limitador-67566c68b4-lbws6

Created

Created container: limitador

kuadrant-system

kubelet

limitador-limitador-67566c68b4-lbws6

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 12.247s (12.247s including waiting). Image size: 249943843 bytes.

kuadrant-system

replicaset-controller

limitador-limitador-64c8f475fb

SuccessfulDelete

Deleted pod: limitador-limitador-64c8f475fb-5ndg6

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled down replica set limitador-limitador-64c8f475fb from 1 to 0

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-5ndg6

Killing

Stopping container limitador

kuadrant-system

kubelet

authorino-68bd676465-5ttpp

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-68bd676465 from 0 to 1

kuadrant-system

multus

authorino-68bd676465-5ttpp

AddedInterface

Add eth0 [10.132.0.27/23] from ovn-kubernetes

kuadrant-system

replicaset-controller

authorino-68bd676465

SuccessfulCreate

Created pod: authorino-68bd676465-5ttpp

kuadrant-system

kubelet

authorino-68bd676465-5ttpp

Started

Started container authorino

kuadrant-system

kubelet

authorino-68bd676465-5ttpp

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 617ms (617ms including waiting). Image size: 199256829 bytes.

kuadrant-system

kubelet

authorino-68bd676465-5ttpp

Created

Created container: authorino

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-79cbc94b89 from 1 to 0

kuadrant-system

kubelet

authorino-79cbc94b89-64sf6

Killing

Stopping container authorino

kuadrant-system

replicaset-controller

authorino-79cbc94b89

SuccessfulDelete

Deleted pod: authorino-79cbc94b89-64sf6

openshift-ingress

replicaset-controller

istiod-openshift-gateway-55ff986f96

SuccessfulCreate

Created pod: istiod-openshift-gateway-55ff986f96-tvvcc

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-55ff986f96 from 0 to 1

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-tvvcc

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8"

openshift-ingress

multus

istiod-openshift-gateway-55ff986f96-tvvcc

AddedInterface

Add eth0 [10.134.0.27/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-tvvcc

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" in 2.488s (2.488s including waiting). Image size: 179322965 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-tvvcc

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-tvvcc

Started

Started container discovery

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7cd77c7ffd

SuccessfulDelete

Deleted pod: istiod-openshift-gateway-7cd77c7ffd-wcfq9

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled down replica set istiod-openshift-gateway-7cd77c7ffd from 1 to 0

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-wcfq9

Killing

Stopping container discovery
(x5)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

deployment-controller

router-gateway-1-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-1-openshift-default-6c59fbf55c from 0 to 1

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

replicaset-controller

router-gateway-1-openshift-default-6c59fbf55c

SuccessfulCreate

Created pod: router-gateway-1-openshift-default-6c59fbf55c-l24v6

kserve-ci-e2e-test

multus

router-gateway-1-openshift-default-6c59fbf55c-l24v6

AddedInterface

Add eth0 [10.132.0.28/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-l24v6

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-l24v6

Started

Started container istio-proxy

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-l24v6

Created

Created container: istio-proxy

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-l24v6

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 2.622s (2.622s including waiting). Image size: 163191192 bytes.

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuredLoadBalancer

Ensured load balancer
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-configmap-ref-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-7f9f4fb44

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-7f9f4fb44 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

AddedInterface

Add eth0 [10.132.0.29/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061"

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-router-scheduler-5b7dd6c67b

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061"

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-5b7dd6c67b from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc
(x14)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-6fa1027a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route" not found

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" in 3.445s (3.445s including waiting). Image size: 299668949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" in 3.211s (3.211s including waiting). Image size: 299668949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 949ms (949ms including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccb from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccb

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.265s (1.265s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-85955b4ccbk5lnz

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 28.614s (28.614s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Started

Started container tokenizer
(x10)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-55f7ae4a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5df

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

kserve-ci-e2e-test

replicaset-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-66b76b54d9

SuccessfulCreate

Created pod: llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

AddedInterface

Add eth0 [10.132.0.30/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Created

Created container: storage-initializer

kserve-ci-e2e-test

deployment-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-66b76b54d9 from 0 to 1

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5df from 0 to 1

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-service
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Unhealthy

Startup probe failed: Get "http://10.132.0.29:8082/healthz": dial tcp 10.132.0.29:8082: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1"

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-5b7ddfpmq2

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7f9f4fb44-tnllj

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" in 26.022s (26.022s including waiting). Image size: 3458932386 bytes.
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-ha-replicas-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f974 from 0 to 2

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

AddedInterface

Add eth0 [10.133.0.51/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-7c644fd568

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f974

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f974

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-7c644fd568 from 0 to 1

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

AddedInterface

Add eth0 [10.133.0.52/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Created

Created container: storage-initializer
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

AddedInterface

Add eth0 [10.132.0.31/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.253s (1.253s including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4fwgm69

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-7c644fd568-xlr5t

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.512s (31.512s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

precise-prefix-cache-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-7db845475b

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-7db845475b-cn8h6

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-7db845475b from 0 to 2

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-7db845475b

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-7db845475b-rngn7

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-cb5747bc8 from 0 to 1

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-7db845475b-rngn7

AddedInterface

Add eth0 [10.133.0.53/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

AddedInterface

Add eth0 [10.132.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-7db845475b-cn8h6

AddedInterface

Add eth0 [10.132.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc8

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.144s (1.145s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Updated

Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
(x3)

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-7675b4f9kcx7

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.133.0.52:9003" within 1s: context deadline exceeded
(x2)

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Unhealthy

Liveness probe failed: timeout: failed to connect service "10.132.0.33:9003" within 1s: context deadline exceeded
(x9)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Unhealthy

Startup probe failed: Get "https://10.133.0.50:8000/health": dial tcp 10.133.0.50:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-rngn7

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-7db845475b-cn8h6

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-cb5747bc247r8

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-round-trip

Created

Created v1.Deployment kserve-ci-e2e-test/conv-test-round-trip-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-round-trip

Error

Reconciliation failed: failed to reconcile workload: failed to reconcile single node workload: failed to reconcile main workload: failed to get current deployment kserve-ci-e2e-test/conv-test-round-trip-kserve: Deployment.apps "conv-test-round-trip-kserve" not found

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-round-trip

Created

Created v1.Secret kserve-ci-e2e-test/conv-test-round-trip-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

replicaset-controller

conv-test-round-trip-kserve-fb84c864

SuccessfulCreate

Created pod: conv-test-round-trip-kserve-fb84c864-t4g46

kserve-ci-e2e-test

multus

conv-test-round-trip-kserve-fb84c864-t4g46

AddedInterface

Add eth0 [10.133.0.54/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

conv-test-round-trip-kserve

ScalingReplicaSet

Scaled up replica set conv-test-round-trip-kserve-fb84c864 from 0 to 1

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Created

Created container: main

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Started

Started container main

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine
(x2)

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Unhealthy

Startup probe failed: Get "https://10.133.0.54:8000/health": dial tcp 10.133.0.54:8000: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-7db4d9c6b5

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-7db4d9c6b5 from 0 to 1

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-5698dc8fcf from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-5698dc8fcf

SuccessfulCreate

Created pod: stop-feature-test-kserve-5698dc8fcf-zspvz

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

AddedInterface

Add eth0 [10.132.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

stop-feature-test-kserve-5698dc8fcf-zspvz

AddedInterface

Add eth0 [10.133.0.55/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Started

Started container storage-initializer
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Created

Created container: main

kserve-ci-e2e-test

kubelet

conv-test-round-trip-kserve-fb84c864-t4g46

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Updated

Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-67b5b4c5dfxsdgd

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schekzsdr

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-968f7c9f5 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-968f7c9f5

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-968f7c9f5-dcrlb

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Created

Created container: storage-initializer
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

AddedInterface

Add eth0 [10.133.0.56/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-975d74c9f from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-router-scheduler-975d74c9f

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

AddedInterface

Add eth0 [10.132.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs
(x11)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Unhealthy

Startup probe failed: Get "https://10.133.0.55:8000/health": dial tcp 10.133.0.55:8000: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-zspvz

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1alpha2.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-zg9vl

Killing

Stopping container tokenizer
(x25)

kserve-ci-e2e-test

OpenDataHubModelController

stop-feature-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found
(x11)

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Unhealthy

Startup probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-5698dc8fcf from 0 to 1

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-5698dc8fcf

SuccessfulCreate

Created pod: stop-feature-test-kserve-5698dc8fcf-ng9bf

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

AddedInterface

Add eth0 [10.132.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-7db4d9c6b5 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-7db4d9c6b5

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

kserve-ci-e2e-test

multus

stop-feature-test-kserve-5698dc8fcf-ng9bf

AddedInterface

Add eth0 [10.133.0.57/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-968f7c9f5-dcrlb

Killing

Stopping container main
(x2)

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-1

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-975d74c9dwv6x

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-5959bdbb55

SuccessfulCreate

Created pod: router-with-refs-test-kserve-5959bdbb55-8fxgx

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-5959bdbb55 from 0 to 1
(x5)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-5959bdbb55-8fxgx

AddedInterface

Add eth0 [10.133.0.58/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

AddedInterface

Add eth0 [10.132.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-router-scheduler-8657c58775 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-router-scheduler-8657c58775

SuccessfulCreate

Created pod: router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs
(x10)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Unhealthy

Startup probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-5698dc8fcf-ng9bf

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-7db4d9c6b5-jdgbg

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.132.0.36:9003" within 1s: context deadline exceeded
(x10)

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Unhealthy

Startup probe failed: Get "https://10.133.0.58:8000/health": dial tcp 10.133.0.58:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5959bdbb55-8fxgx

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-8657c58775-xld44

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Error

Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to create v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs: client rate limiter Wait returned an error: context canceled
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-05aa9bba

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-rb

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fb85477

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6b

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fb85477 from 0 to 1

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6b from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-9bc8bb4c7

SuccessfulCreate

Created pod: llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

kserve-ci-e2e-test

deployment-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-9bc8bb4c7 from 0 to 1

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

AddedInterface

Add eth0 [10.132.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

AddedInterface

Add eth0 [10.133.0.60/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

AddedInterface

Add eth0 [10.133.0.59/23] from ovn-kubernetes
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 913ms (913ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-02ac88e7

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

AddedInterface

Add eth0 [10.133.0.61/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs
(x16)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Unhealthy

Startup probe failed: Get "https://10.133.0.59:8001/health": dial tcp 10.133.0.59:8001: connect: connection refused
(x16)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Unhealthy

Startup probe failed: Get "https://10.133.0.60:8000/health": dial tcp 10.133.0.60:8000: connect: connection refused
(x15)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Unhealthy

Startup probe failed: Get "https://10.133.0.61:8000/health": dial tcp 10.133.0.61:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schew2pwj

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-8fjng27

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-5785784b6bjlf5l

Killing

Stopping container llm-d-routing-sidecar
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-pd-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-855cffc847

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-57c68f4f4 from 0 to 1

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-855cffc847 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-57f6888d64 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-prefill-57f6888d64

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

AddedInterface

Add eth0 [10.132.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Pulled

Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-router-scheduler-57c68f4f4

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

AddedInterface

Add eth0 [10.133.0.63/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

AddedInterface

Add eth0 [10.133.0.62/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-848f6f45d5fzz22

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-scc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Started

Started container storage-initializer

kserve-ci-e2e-test

deployment-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-7dd8c7bc9 from 0 to 1

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container storage-initializer

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 successful

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

AddedInterface

Add eth0 [10.133.0.64/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Created

Created container: storage-initializer

kserve-ci-e2e-test

replicaset-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-7dd8c7bc9

SuccessfulCreate

Created pod: llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

AddedInterface

Add eth0 [10.132.0.41/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Started

Started container storage-initializer

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

CreatingRevision

Creating revision with key 58767f6b5b for a newly created LeaderWorkerSet

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

AddedInterface

Add eth0 [10.132.0.40/23] from ovn-kubernetes
(x2)

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Replicas are progressing, with 0 groups ready of total 1 groups
(x15)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-f312f5ec

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route" not found

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created leader statefulset llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn successful
(x8)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1"

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" in 43.931s (43.931s including waiting). Image size: 3458932386 bytes.

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: main
(x16)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Unhealthy

Startup probe failed: Get "https://10.133.0.63:8000/health": dial tcp 10.133.0.63:8000: connect: connection refused
(x16)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Unhealthy

Startup probe failed: Get "https://10.133.0.62:8001/health": dial tcp 10.133.0.62:8001: connect: connection refused
(x15)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Unhealthy

Startup probe failed: Get "https://10.133.0.64:8000/health": dial tcp 10.133.0.64:8000: connect: connection refused
(x2)

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

AllGroupsReady

All replicas are ready, with 1 groups ready of total 1 groups

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-57c68s66lc

Killing

Stopping container tokenizer

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-2

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-2

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-57f6888d64-vq5hz

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-855cffc847-fzvhl

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

deployment-controller

router-gateway-2-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-2-openshift-default-6866b85949 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

router-gateway-2-openshift-default-6866b85949

SuccessfulCreate

Created pod: router-gateway-2-openshift-default-6866b85949-lflmt

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-lflmt

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"

kserve-ci-e2e-test

multus

router-gateway-2-openshift-default-6866b85949-lflmt

AddedInterface

Add eth0 [10.134.0.29/23] from ovn-kubernetes

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuredLoadBalancer

Ensured load balancer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-7f8ccb594c

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-prefill-64c46b7dfb from 0 to 1

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-7f8ccb594c from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-prefill-64c46b7dfb

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

AddedInterface

Add eth0 [10.133.0.65/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-8546c8d74b from 0 to 1

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

AddedInterface

Add eth0 [10.132.0.42/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-router-scheduler-8546c8d74b

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Started

Started container storage-initializer
(x9)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-lflmt

Created

Created container: istio-proxy

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-lflmt

Started

Started container istio-proxy

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 767ms (767ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-lflmt

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 9.457s (9.457s including waiting). Image size: 163191192 bytes.

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

AddedInterface

Add eth0 [10.132.0.43/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Created

Created container: tokenizer
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-inline-config-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found
(x3)

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-lflmt

Unhealthy

Startup probe failed: Get "http://10.134.0.29:15021/healthz/ready": dial tcp 10.134.0.29:15021: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.17.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Started

Started container main

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-587498644

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-587498644-znkwq

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-587498644 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-99db9974 from 0 to 1

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" already present on machine

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

AddedInterface

Add eth0 [10.133.0.66/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-router-scheduler-99db9974

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061"

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-587498644-znkwq

AddedInterface

Add eth0 [10.134.0.30/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Unhealthy

Readiness probe failed: Get "http://10.132.0.41:8082/healthz": dial tcp 10.132.0.41:8082: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schebsshf

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.132.0.41:9003" within 1s: context deadline exceeded

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ab565fe3972fd5cd2929b41b0a6a9face626513b818fecb7a38edcdfe0857061" in 4.638s (4.638s including waiting). Image size: 299668949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.212s (1.212s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-587498644-znkwq

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-99db9zx4rn

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
(x9)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Unhealthy

Startup probe failed: Get "https://10.133.0.65:8000/health": dial tcp 10.133.0.65:8000: connect: connection refused
(x9)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Unhealthy

Startup probe failed: Get "https://10.132.0.42:8001/health": dial tcp 10.132.0.42:8001: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Unhealthy

Readiness probe failed: Get "http://10.132.0.43:8082/healthz": dial tcp 10.132.0.43:8082: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-8546c8d7447ksk

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-7f8ccb594c-zf7sz

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-64c46b7dfb-d9lld

Killing

Stopping container main