Time Namespace Component RelatedObject Reason Message

kserve-ci-e2e-test

conv-test-lora-crit-kserve-568c94495b-pmqkp

Scheduled

Successfully assigned kserve-ci-e2e-test/conv-test-lora-crit-kserve-568c94495b-pmqkp to ip-10-0-133-36.ec2.internal

kuadrant-system

kuadrant-console-plugin-6c886788f8-bvg5d

Scheduled

Successfully assigned kuadrant-system/kuadrant-console-plugin-6c886788f8-bvg5d to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-d6b549768-lcd6s to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc to ip-10-0-142-242.ec2.internal

openshift-ingress

router-default-7f4c79d4bd-dxrjx

Scheduled

Successfully assigned openshift-ingress/router-default-7f4c79d4bd-dxrjx to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9 to ip-10-0-142-242.ec2.internal

kuadrant-system

limitador-limitador-64c8f475fb-z4h84

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-64c8f475fb-z4h84 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

router-with-refs-test-kserve-5d74d49756-cw8kg

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5d74d49756-cw8kg to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-8446b959c6-fxpwp

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-8446b959c6-fxpwp to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-8446b959c6-jb8f8

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-8446b959c6-jb8f8 to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2 to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-659d8476f4-2gsxl to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-659d8476f4-f8gx9 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Scheduled

Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 to ip-10-0-133-36.ec2.internal

kuadrant-system

dns-operator-controller-manager-844548ff4c-6kpsg

Scheduled

Successfully assigned kuadrant-system/dns-operator-controller-manager-844548ff4c-6kpsg to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

router-gateway-1-openshift-default-6c59fbf55c-ngwj6

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-ngwj6 to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

router-gateway-2-openshift-default-6866b85949-z7zjj

Scheduled

Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-z7zjj to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f6d44c486-h2n7s to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk to ip-10-0-142-242.ec2.internal

openshift-ingress

router-default-7f4c79d4bd-dxrjx

FailedScheduling

0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

openshift-ingress

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-qkhkg to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Scheduled

Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 to ip-10-0-142-242.ec2.internal

kuadrant-system

limitador-limitador-67566c68b4-mrc5l

Scheduled

Successfully assigned kuadrant-system/limitador-limitador-67566c68b4-mrc5l to ip-10-0-133-36.ec2.internal

kuadrant-system

limitador-operator-controller-manager-c7fb4c8d5-grrpw

Scheduled

Successfully assigned kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-grrpw to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8 to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj to ip-10-0-133-36.ec2.internal

kuadrant-system

authorino-operator-7587b89b76-4d9ct

Scheduled

Successfully assigned kuadrant-system/authorino-operator-7587b89b76-4d9ct to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw to ip-10-0-134-232.ec2.internal

kuadrant-system

authorino-79cbc94b89-j86ds

Scheduled

Successfully assigned kuadrant-system/authorino-79cbc94b89-j86ds to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

Scheduled

Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p to ip-10-0-142-242.ec2.internal

kuadrant-system

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

Scheduled

Successfully assigned kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-z9p66 to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Scheduled

Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp to ip-10-0-134-232.ec2.internal

openshift-ingress

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Scheduled

Successfully assigned openshift-ingress/openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d to ip-10-0-142-242.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Scheduled

Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl to ip-10-0-142-242.ec2.internal

kuadrant-system

authorino-68bd676465-jf4qz

Scheduled

Successfully assigned kuadrant-system/authorino-68bd676465-jf4qz to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Scheduled

Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb to ip-10-0-134-232.ec2.internal

kuadrant-system

authorino-674b59b84c-ckdb9

Scheduled

Successfully assigned kuadrant-system/authorino-674b59b84c-ckdb9 to ip-10-0-134-232.ec2.internal

kserve-ci-e2e-test

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Scheduled

Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c6c7c46cb-clzrk to ip-10-0-133-36.ec2.internal

kserve-ci-e2e-test

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Scheduled

Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx to ip-10-0-133-36.ec2.internal

openshift-ingress

istiod-openshift-gateway-55ff986f96-nzp8n

Scheduled

Successfully assigned openshift-ingress/istiod-openshift-gateway-55ff986f96-nzp8n to ip-10-0-142-242.ec2.internal

openshift-ingress

deployment-controller

router-default

ScalingReplicaSet

Scaled up replica set router-default-7f4c79d4bd from 0 to 1

openshift-ingress

service-controller

router-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

service-controller

router-default

UnAvailableLoadBalancer

There are no available nodes for LoadBalancer
(x9)

openshift-ingress

replicaset-controller

router-default-7f4c79d4bd

FailedCreate

Error creating: pods "router-default-7f4c79d4bd-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

openshift-ingress

service-controller

router-default

EnsuredLoadBalancer

Ensured load balancer
(x15)

openshift-ingress

replicaset-controller

router-default-7f4c79d4bd

FailedCreate

Error creating: pods "router-default-7f4c79d4bd-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the cluster does not have any nodes

openshift-ingress

replicaset-controller

router-default-7f4c79d4bd

SuccessfulCreate

Created pod: router-default-7f4c79d4bd-dxrjx
(x3)

openshift-ingress

service-controller

router-default

UpdatedLoadBalancer

Updated load balancer with new hosts
(x6)

openshift-ingress

kubelet

router-default-7f4c79d4bd-dxrjx

FailedMount

MountVolume.SetUp failed for volume "metrics-certs" : secret "router-metrics-certs-default" not found
(x6)

openshift-ingress

kubelet

router-default-7f4c79d4bd-dxrjx

FailedMount

MountVolume.SetUp failed for volume "service-ca-bundle" : configmap references non-existent config key: service-ca.crt

openshift-ingress

kubelet

router-default-7f4c79d4bd-dxrjx

Started

Started container router

openshift-ingress

kubelet

router-default-7f4c79d4bd-dxrjx

Created

Created container: router

openshift-ingress

kubelet

router-default-7f4c79d4bd-dxrjx

Pulled

Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:294af5c64228434d1ed6ee8ea3ac802e3c999aa847223e3b2efa18425a9fe421" already present on machine

openshift-ingress

multus

router-default-7f4c79d4bd-dxrjx

AddedInterface

Add eth0 [10.132.0.9/23] from ovn-kubernetes

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-7cd77c7ffd from 0 to 1

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7cd77c7ffd

SuccessfulCreate

Created pod: istiod-openshift-gateway-7cd77c7ffd-qkhkg

openshift-ingress

multus

istiod-openshift-gateway-7cd77c7ffd-qkhkg

AddedInterface

Add eth0 [10.133.0.33/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8"

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Started

Started container discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Created

Created container: discovery

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" in 2.538s (2.538s including waiting). Image size: 179322965 bytes.

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

ProbeError

Readiness probe error: HTTP probe failed with statuscode: 503 body:

openshift-ingress

multus

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

AddedInterface

Add eth0 [10.133.0.34/23] from ovn-kubernetes

openshift-ingress

replicaset-controller

openshift-ai-inference-openshift-default-6b94bb86d8

SuccessfulCreate

Created pod: openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

openshift-ingress

gateway_labeler_controller

openshift-ai-inference

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway openshift-ai-inference

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Pulling

Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe"

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

openshift-ingress

deployment-controller

openshift-ai-inference-openshift-default

ScalingReplicaSet

Scaled up replica set openshift-ai-inference-openshift-default-6b94bb86d8 from 0 to 1

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Created

Created container: istio-proxy

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Started

Started container istio-proxy

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Pulled

Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" in 2.704s (2.704s including waiting). Image size: 163191192 bytes.

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

Unhealthy

Startup probe failed: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused

openshift-ingress

kubelet

openshift-ai-inference-openshift-default-6b94bb86d8-q7m5d

ProbeError

Startup probe error: Get "http://10.133.0.34:15021/healthz/ready": dial tcp 10.133.0.34:15021: connect: connection refused body:

openshift-ingress

service-controller

openshift-ai-inference-openshift-default

EnsuredLoadBalancer

Ensured load balancer

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

replicaset-controller

dns-operator-controller-manager-844548ff4c

SuccessfulCreate

Created pod: dns-operator-controller-manager-844548ff4c-6kpsg

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallWaiting

installing: waiting for deployment dns-operator-controller-manager to become ready: deployment "dns-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6kpsg

Pulling

Pulling image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf"

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

deployment-controller

dns-operator-controller-manager

ScalingReplicaSet

Scaled up replica set dns-operator-controller-manager-844548ff4c from 0 to 1

kuadrant-system

multus

dns-operator-controller-manager-844548ff4c-6kpsg

AddedInterface

Add eth0 [10.133.0.39/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

deployment-controller

limitador-operator-controller-manager

ScalingReplicaSet

Scaled up replica set limitador-operator-controller-manager-c7fb4c8d5 from 0 to 1

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallWaiting

installing: waiting for deployment limitador-operator-controller-manager to become ready: deployment "limitador-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

replicaset-controller

limitador-operator-controller-manager-c7fb4c8d5

SuccessfulCreate

Created pod: limitador-operator-controller-manager-c7fb4c8d5-grrpw

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6kpsg

Started

Started container manager

kuadrant-system

dns-operator-controller-manager-844548ff4c-6kpsg_a8756d10-7fc4-4759-801a-40ba546d3eab

a3f98d6c.kuadrant.io

LeaderElection

dns-operator-controller-manager-844548ff4c-6kpsg_a8756d10-7fc4-4759-801a-40ba546d3eab became leader

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6kpsg

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:b4e7ba67509320ca9ac5d63cc4add987fad05b098c4a7cd8dd91f264731177cf" in 2.622s (2.622s including waiting). Image size: 186790172 bytes.

kuadrant-system

kubelet

dns-operator-controller-manager-844548ff4c-6kpsg

Created

Created container: manager

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-grrpw

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66"

kuadrant-system

multus

limitador-operator-controller-manager-c7fb4c8d5-grrpw

AddedInterface

Add eth0 [10.133.0.40/23] from ovn-kubernetes

kuadrant-system

deployment-controller

authorino-operator

ScalingReplicaSet

Scaled up replica set authorino-operator-7587b89b76 from 0 to 1

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallWaiting

installing: waiting for deployment authorino-operator to become ready: deployment "authorino-operator" not available: Deployment does not have minimum availability.

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-grrpw

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:9a3ccb54bc904521277b87e1d178d8134cc6ef744eb4ec5b64177355fd5a9e66" in 1.878s (1.878s including waiting). Image size: 175506011 bytes.

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-grrpw

Created

Created container: manager

kuadrant-system

kubelet

limitador-operator-controller-manager-c7fb4c8d5-grrpw

Started

Started container manager

kuadrant-system

kubelet

authorino-operator-7587b89b76-4d9ct

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733"

kuadrant-system

multus

authorino-operator-7587b89b76-4d9ct

AddedInterface

Add eth0 [10.134.0.25/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

replicaset-controller

authorino-operator-7587b89b76

SuccessfulCreate

Created pod: authorino-operator-7587b89b76-4d9ct

kuadrant-system

limitador-operator-controller-manager-c7fb4c8d5-grrpw_ca6ad486-099c-4d81-a0e5-3a7e2cbf7bf6

3745a16e.kuadrant.io

LeaderElection

limitador-operator-controller-manager-c7fb4c8d5-grrpw_ca6ad486-099c-4d81-a0e5-3a7e2cbf7bf6 became leader

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

RequirementsUnknown

requirements not yet checked

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

RequirementsNotMet

one or more requirements couldn't be found

kuadrant-system

kubelet

authorino-operator-7587b89b76-4d9ct

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:4e581c0172549df72c6ef8978e96d771645bc36298d2ccf9240af4e71c088733" in 2.727s (2.727s including waiting). Image size: 161991975 bytes.

kuadrant-system

replicaset-controller

kuadrant-operator-controller-manager-6ddf9554fc

SuccessfulCreate

Created pod: kuadrant-operator-controller-manager-6ddf9554fc-z9p66

kuadrant-system

authorino-operator-7587b89b76-4d9ct_b9c1ce3b-1ea0-48a9-af47-8e71dae0c377

aac3a15d.authorino.kuadrant.io

LeaderElection

authorino-operator-7587b89b76-4d9ct_b9c1ce3b-1ea0-48a9-af47-8e71dae0c377 became leader

kuadrant-system

kubelet

authorino-operator-7587b89b76-4d9ct

Started

Started container manager

kuadrant-system

kubelet

authorino-operator-7587b89b76-4d9ct

Created

Created container: manager

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

AllRequirementsMet

all requirements found, attempting install

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallSucceeded

waiting for install components to report healthy

kuadrant-system

deployment-controller

kuadrant-operator-controller-manager

ScalingReplicaSet

Scaled up replica set kuadrant-operator-controller-manager-6ddf9554fc from 0 to 1

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8"

kuadrant-system

multus

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

AddedInterface

Add eth0 [10.133.0.41/23] from ovn-kubernetes

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallWaiting

installing: waiting for deployment kuadrant-operator-controller-manager to become ready: deployment "kuadrant-operator-controller-manager" not available: Deployment does not have minimum availability.

kuadrant-system

kuadrant-operator-controller-manager-6ddf9554fc-z9p66_a17ac8ac-9801-4dbd-9050-d9e9c9a17f83

f139389e.kuadrant.io

LeaderElection

kuadrant-operator-controller-manager-6ddf9554fc-z9p66_a17ac8ac-9801-4dbd-9050-d9e9c9a17f83 became leader

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

Created

Created container: manager

kuadrant-system

operator-lifecycle-manager

dns-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

Started

Started container manager

kuadrant-system

kubelet

kuadrant-operator-controller-manager-6ddf9554fc-z9p66

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:7aa6d6dbbde488260789adb94fa232a06d1e635cc2aaf86af24f8cf14a17eef8" in 5.781s (5.781s including waiting). Image size: 411782238 bytes.

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-bvg5d

FailedMount

MountVolume.SetUp failed for volume "plugin-serving-cert" : secret "plugin-serving-cert" not found

kuadrant-system

operator-lifecycle-manager

limitador-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

deployment-controller

kuadrant-console-plugin

ScalingReplicaSet

Scaled up replica set kuadrant-console-plugin-6c886788f8 from 0 to 1

kuadrant-system

replicaset-controller

kuadrant-console-plugin-6c886788f8

SuccessfulCreate

Created pod: kuadrant-console-plugin-6c886788f8-bvg5d

kuadrant-system

multus

kuadrant-console-plugin-6c886788f8-bvg5d

AddedInterface

Add eth0 [10.132.0.23/23] from ovn-kubernetes

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-bvg5d

Pulling

Pulling image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57"

kuadrant-system

operator-lifecycle-manager

authorino-operator.v1.3.0

InstallSucceeded

install strategy completed with no errors

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-bvg5d

Started

Started container kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-bvg5d

Created

Created container: kuadrant-console-plugin

kuadrant-system

kubelet

kuadrant-console-plugin-6c886788f8-bvg5d

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/rhcl-console-plugin-rhel9@sha256:41755d5e16ee5409cfe8888cf5883b262820baea4eae449fcf61f06ab1756f57" in 4.527s (4.527s including waiting). Image size: 339541674 bytes.

kuadrant-system

operator-lifecycle-manager

rhcl-operator.v1.3.2

InstallSucceeded

install strategy completed with no errors

kuadrant-system

replicaset-controller

limitador-limitador-64c8f475fb

SuccessfulCreate

Created pod: limitador-limitador-64c8f475fb-z4h84

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-64c8f475fb from 0 to 1

kuadrant-system

replicaset-controller

authorino-674b59b84c

SuccessfulCreate

Created pod: authorino-674b59b84c-ckdb9

kuadrant-system

kubelet

limitador-limitador-67566c68b4-mrc5l

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-z4h84

Pulling

Pulling image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0"

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-79cbc94b89 from 0 to 1

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-674b59b84c from 0 to 1

kuadrant-system

multus

limitador-limitador-64c8f475fb-z4h84

AddedInterface

Add eth0 [10.134.0.26/23] from ovn-kubernetes

kuadrant-system

replicaset-controller

authorino-79cbc94b89

SuccessfulCreate

Created pod: authorino-79cbc94b89-j86ds

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled up replica set limitador-limitador-67566c68b4 from 0 to 1

kuadrant-system

replicaset-controller

limitador-limitador-67566c68b4

SuccessfulCreate

Created pod: limitador-limitador-67566c68b4-mrc5l

kuadrant-system

multus

limitador-limitador-67566c68b4-mrc5l

AddedInterface

Add eth0 [10.132.0.24/23] from ovn-kubernetes

kuadrant-system

multus

authorino-79cbc94b89-j86ds

AddedInterface

Add eth0 [10.132.0.25/23] from ovn-kubernetes

kuadrant-system

kubelet

limitador-limitador-67566c68b4-mrc5l

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 1.139s (1.139s including waiting). Image size: 249943843 bytes.

kuadrant-system

kubelet

limitador-limitador-67566c68b4-mrc5l

Created

Created container: limitador

kuadrant-system

kubelet

limitador-limitador-67566c68b4-mrc5l

Started

Started container limitador

kuadrant-system

kubelet

authorino-79cbc94b89-j86ds

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

kubelet

authorino-674b59b84c-ckdb9

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

multus

authorino-674b59b84c-ckdb9

AddedInterface

Add eth0 [10.134.0.27/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-79cbc94b89-j86ds

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 2.432s (2.432s including waiting). Image size: 199256829 bytes.

kuadrant-system

kubelet

authorino-79cbc94b89-j86ds

Created

Created container: authorino

kuadrant-system

kubelet

authorino-79cbc94b89-j86ds

Started

Started container authorino

kuadrant-system

replicaset-controller

authorino-674b59b84c

SuccessfulDelete

Deleted pod: authorino-674b59b84c-ckdb9

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-674b59b84c from 1 to 0

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-z4h84

Started

Started container limitador

kuadrant-system

kubelet

authorino-674b59b84c-ckdb9

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 4.33s (4.33s including waiting). Image size: 199256829 bytes.

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-z4h84

Created

Created container: limitador

kuadrant-system

kubelet

authorino-674b59b84c-ckdb9

Killing

Stopping container authorino

kuadrant-system

kubelet

authorino-674b59b84c-ckdb9

Started

Started container authorino

kuadrant-system

kubelet

authorino-674b59b84c-ckdb9

Created

Created container: authorino

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-z4h84

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/limitador-rhel9@sha256:43bc4021bb30c7ef305e01c11e22ca5584c7e2c192b4db0a5646f04f715ee3b0" in 5.264s (5.264s including waiting). Image size: 249943843 bytes.

kuadrant-system

deployment-controller

limitador-limitador

ScalingReplicaSet

Scaled down replica set limitador-limitador-64c8f475fb from 1 to 0

kuadrant-system

replicaset-controller

limitador-limitador-64c8f475fb

SuccessfulDelete

Deleted pod: limitador-limitador-64c8f475fb-z4h84

kuadrant-system

kubelet

limitador-limitador-64c8f475fb-z4h84

Killing

Stopping container limitador

kuadrant-system

replicaset-controller

authorino-68bd676465

SuccessfulCreate

Created pod: authorino-68bd676465-jf4qz

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled up replica set authorino-68bd676465 from 0 to 1

kuadrant-system

kubelet

authorino-68bd676465-jf4qz

Created

Created container: authorino

kuadrant-system

multus

authorino-68bd676465-jf4qz

AddedInterface

Add eth0 [10.134.0.28/23] from ovn-kubernetes

kuadrant-system

kubelet

authorino-68bd676465-jf4qz

Pulling

Pulling image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8"

kuadrant-system

kubelet

authorino-68bd676465-jf4qz

Started

Started container authorino

kuadrant-system

kubelet

authorino-68bd676465-jf4qz

Pulled

Successfully pulled image "registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c2208382e16c501e4ed58aea83f54d108567f36853cf79f7ffdf8ee54ff4ace8" in 465ms (465ms including waiting). Image size: 199256829 bytes.

kuadrant-system

deployment-controller

authorino

ScalingReplicaSet

Scaled down replica set authorino-79cbc94b89 from 1 to 0

kuadrant-system

replicaset-controller

authorino-79cbc94b89

SuccessfulDelete

Deleted pod: authorino-79cbc94b89-j86ds

kuadrant-system

kubelet

authorino-79cbc94b89-j86ds

Killing

Stopping container authorino

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-nzp8n

Started

Started container discovery

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled up replica set istiod-openshift-gateway-55ff986f96 from 0 to 1

openshift-ingress

multus

istiod-openshift-gateway-55ff986f96-nzp8n

AddedInterface

Add eth0 [10.133.0.42/23] from ovn-kubernetes

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-nzp8n

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:028e10651db0d1ddb769a27c9483c6d41be6ac597f253afd9d599f395d9c82d8" already present on machine

openshift-ingress

kubelet

istiod-openshift-gateway-55ff986f96-nzp8n

Created

Created container: discovery

openshift-ingress

replicaset-controller

istiod-openshift-gateway-55ff986f96

SuccessfulCreate

Created pod: istiod-openshift-gateway-55ff986f96-nzp8n

openshift-ingress

kubelet

istiod-openshift-gateway-7cd77c7ffd-qkhkg

Killing

Stopping container discovery

openshift-ingress

replicaset-controller

istiod-openshift-gateway-7cd77c7ffd

SuccessfulDelete

Deleted pod: istiod-openshift-gateway-7cd77c7ffd-qkhkg

openshift-ingress

deployment-controller

istiod-openshift-gateway

ScalingReplicaSet

Scaled down replica set istiod-openshift-gateway-7cd77c7ffd from 1 to 0
(x6)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x6)

openshift-ingress

horizontal-pod-autoscaler

istiod-openshift-gateway

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-ngwj6

Created

Created container: istio-proxy

kserve-ci-e2e-test

deployment-controller

router-gateway-1-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-1-openshift-default-6c59fbf55c from 0 to 1

kserve-ci-e2e-test

multus

router-gateway-1-openshift-default-6c59fbf55c-ngwj6

AddedInterface

Add eth0 [10.133.0.46/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-ngwj6

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" already present on machine

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

kubelet

router-gateway-1-openshift-default-6c59fbf55c-ngwj6

Started

Started container istio-proxy

kserve-ci-e2e-test

replicaset-controller

router-gateway-1-openshift-default-6c59fbf55c

SuccessfulCreate

Created pod: router-gateway-1-openshift-default-6c59fbf55c-ngwj6

kserve-ci-e2e-test

service-controller

router-gateway-1-openshift-default

EnsuredLoadBalancer

Ensured load balancer

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs

kserve-ci-e2e-test

replicaset-controller

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cf

SuccessfulCreate

Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097"

kserve-ci-e2e-test

multus

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

AddedInterface

Add eth0 [10.133.0.47/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route

kserve-ci-e2e-test

deployment-controller

gw-section-name-router-with-gat-2f0a622e-kserve

ScalingReplicaSet

Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cf from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-2f0a622e

Created

Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" in 3.668s (3.668s including waiting). Image size: 299845049 bytes.

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Started

Started container storage-initializer
(x14)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-inline-config-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Created

Created container: main

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Started

Started container main

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.116s (1.116s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-d6b549768

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-d6b549768-lcd6s

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-d6b549768 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-d6b549768-lcd6s

AddedInterface

Add eth0 [10.132.0.27/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097"

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc

kserve-ci-e2e-test

replicaset-controller

scheduler-inline-config-test-kserve-router-scheduler-5ff4468f86

SuccessfulCreate

Created pod: scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

kserve-ci-e2e-test

multus

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

AddedInterface

Add eth0 [10.134.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097"

kserve-ci-e2e-test

deployment-controller

scheduler-inline-config-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-5ff4468f86 from 0 to 1

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-2f0a622e-kserve-7668df57cfn4lgl

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" in 3.303s (3.303s including waiting). Image size: 299845049 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" in 2.781s (2.781s including waiting). Image size: 299845049 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.012s (1.012s including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Started

Started container main

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4

SuccessfulCreate

Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

kserve-ci-e2e-test

multus

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

gw-section-name-router-with-gat-f1d92d0f-kserve

ScalingReplicaSet

Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

gw-section-name-router-with-gat-f1d92d0f

Created

Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route

kserve-ci-e2e-test

kubelet

gw-section-name-router-with-gat-f1d92d0f-kserve-6bfb88bbd4zdg9p

Killing

Stopping container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-6fa1027a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route" not found

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Created

Created container: storage-initializer

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76 from 0 to 1

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

AddedInterface

Add eth0 [10.132.0.28/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-route

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 29.895s (29.895s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Unhealthy

Startup probe failed: Get "http://10.134.0.32:8082/healthz": dial tcp 10.134.0.32:8082: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-6fa1027a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 967ms (967ms including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-router-scheduler-5ff44t5nwp

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-inline-config-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb
(x3)

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

FailedMount

MountVolume.SetUp failed for volume "tls-certs" : secret "scheduler-inline-config-test-kserve-self-signed-certs" not found

kserve-ci-e2e-test

kubelet

scheduler-inline-config-test-kserve-d6b549768-lcd6s

Killing

Stopping container main
(x10)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-configmap-ref-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-7ffc767776 from 0 to 1

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

AddedInterface

Add eth0 [10.134.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-router-scheduler-7477559468

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa

kserve-ci-e2e-test

multus

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

deployment-controller

scheduler-configmap-ref-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-7477559468 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

scheduler-configmap-ref-test-kserve-7ffc767776

SuccessfulCreate

Created pod: scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Started

Started container tokenizer
(x2)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

Killing

Stopping container main
(x2)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-5ddc4dcf76f29tx

FailedMount

MountVolume.SetUp failed for volume "tls-certs" : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-55f7ae4a

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-route" not found

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-rb

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Started

Started container storage-initializer
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-service

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

AddedInterface

Add eth0 [10.132.0.29/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-798dd76b45

SuccessfulCreate

Created pod: llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

AddedInterface

Add eth0 [10.134.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-scheduler-798dd76b45 from 0 to 1

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-router-scheduler-74775wbm7v

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-configmap-ref-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-configmap-ref-test-kserve-7ffc767776-r8gp9

Killing

Stopping container main
(x14)

kserve-ci-e2e-test

OpenDataHubModelController

scheduler-ha-replicas-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.279s (27.279s including waiting). Image size: 3531177328 bytes.

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-69bd5f69cb from 0 to 1

kserve-ci-e2e-test

deployment-controller

scheduler-ha-replicas-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc996b from 0 to 2

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

AddedInterface

Add eth0 [10.134.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc996b

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-69bd5f69cb

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc996b

SuccessfulCreate

Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

AddedInterface

Add eth0 [10.132.0.30/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.367s (1.367s including waiting). Image size: 87241021 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Started

Started container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Pulling

Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1"

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Created

Created container: main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Updated

Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

scheduler-ha-replicas-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc95ndvf

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-69bd5f69cb-gf8gc

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 32.204s (32.204s including waiting). Image size: 2907020949 bytes.

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Unhealthy

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

precise-prefix-cache-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-659d8476f4

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-659d8476f4-f8gx9

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-659d8476f4

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-659d8476f4-2gsxl

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-659d8476f4 from 0 to 2

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

deployment-controller

precise-prefix-cache-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-6fc856f898 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

precise-prefix-cache-test-kserve-router-scheduler-6fc856f898

SuccessfulCreate

Created pod: precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

AddedInterface

Add eth0 [10.133.0.51/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

AddedInterface

Add eth0 [10.134.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

AddedInterface

Add eth0 [10.134.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Created

Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Created

Created container: tokenizer
(x3)

kserve-ci-e2e-test

kubelet

scheduler-ha-replicas-test-kserve-router-scheduler-76fdcc9xs7nt

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.132.0.30:9003" within 1s: context deadline exceeded
(x2)

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Unhealthy

Liveness probe failed: timeout: failed to connect service "10.134.0.37:9003" within 1s: context deadline exceeded

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Updated

Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs
(x10)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Unhealthy

Startup probe failed: Get "https://10.132.0.29:8000/health": dial tcp 10.132.0.29:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schex6ctw

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-59d97cbbd8zvq5j

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-55f7ae4a

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-teste2c0c7deeb42f8b9846a7a45d2caecd2-llmisvc-mode
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-7c6c7c46cb from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Created

Created container: storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

AddedInterface

Add eth0 [10.132.0.31/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-657d5b7c67 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-router-scheduler-657d5b7c67

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Started

Started container storage-initializer

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-test-kserve-7c6c7c46cb

SuccessfulCreate

Created pod: custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

kserve-ci-e2e-test

multus

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

AddedInterface

Add eth0 [10.132.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Pulling

Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes.

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Created

Created container: main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Started

Started container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-f8gx9

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-659d8476f4-2gsxl

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

precise-prefix-cache-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

precise-prefix-cache-test-kserve-router-scheduler-6fc856f85q2r2

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.134.0.37:9003" within 1s: context deadline exceeded

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-lora-crit

Created

Created v1.HTTPRoute kserve-ci-e2e-test/conv-test-lora-crit-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-lora-crit

Created

Created v1.Service kserve-ci-e2e-test/conv-test-lora-crit-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-lora-crit

Created

Created v1.Secret kserve-ci-e2e-test/conv-test-lora-crit-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-lora-crit

Created

Created v1.Deployment kserve-ci-e2e-test/conv-test-lora-crit-kserve

kserve-ci-e2e-test

replicaset-controller

conv-test-lora-crit-kserve-568c94495b

SuccessfulCreate

Created pod: conv-test-lora-crit-kserve-568c94495b-pmqkp

kserve-ci-e2e-test

deployment-controller

conv-test-lora-crit-kserve

ScalingReplicaSet

Scaled up replica set conv-test-lora-crit-kserve-568c94495b from 0 to 1
(x2)

kserve-ci-e2e-test

kubelet

conv-test-lora-crit-kserve-568c94495b-pmqkp

Started

Started container storage-initializer
(x2)

kserve-ci-e2e-test

kubelet

conv-test-lora-crit-kserve-568c94495b-pmqkp

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine
(x2)

kserve-ci-e2e-test

kubelet

conv-test-lora-crit-kserve-568c94495b-pmqkp

Created

Created container: storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

conv-test-lora-crit

Created

Created v1.DestinationRule kserve-ci-e2e-test/conv-test-lora-crit-kserve-workload-svc

kserve-ci-e2e-test

multus

conv-test-lora-crit-kserve-568c94495b-pmqkp

AddedInterface

Add eth0 [10.132.0.33/23] from ovn-kubernetes
(x2)

kserve-ci-e2e-test

kubelet

conv-test-lora-crit-kserve-568c94495b-pmqkp

BackOff

Back-off restarting failed container storage-initializer in pod conv-test-lora-crit-kserve-568c94495b-pmqkp_kserve-ci-e2e-test(ff901051-be5e-4601-9795-026c725b8317)

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-8446b959c6 from 0 to 1

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-67d85d6d77 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-67d85d6d77

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-8446b959c6

SuccessfulCreate

Created pod: stop-feature-test-kserve-8446b959c6-fxpwp

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

stop-feature-test-kserve-8446b959c6-fxpwp

AddedInterface

Add eth0 [10.132.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine
(x6)

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

AddedInterface

Add eth0 [10.134.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Updated

Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs
(x10)

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Unhealthy

Startup probe failed: Get "https://10.132.0.31:8000/health": dial tcp 10.132.0.31:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-router-scheduler-657d5b7c4x8h2

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-test-kserve-7c6c7c46cb-clzrk

Killing

Stopping container main
(x4)

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-1

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-1

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-5d74d49756 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-5d74d49756

SuccessfulCreate

Created pod: router-with-refs-test-kserve-5d74d49756-cw8kg

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Started

Started container storage-initializer

kserve-ci-e2e-test

deployment-controller

router-with-refs-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-test-kserve-router-scheduler-79b8cd6b7f from 0 to 1

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-5d74d49756-cw8kg

AddedInterface

Add eth0 [10.132.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Started

Started container storage-initializer
(x5)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Created

Created container: storage-initializer

kserve-ci-e2e-test

replicaset-controller

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f

SuccessfulCreate

Created pod: router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb

kserve-ci-e2e-test

multus

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

AddedInterface

Add eth0 [10.134.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine
(x10)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Unhealthy

Startup probe failed: Get "https://10.132.0.34:8000/health": dial tcp 10.132.0.34:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1alpha2.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-zlsr8

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-fxpwp

Killing

Stopping container main

kserve-ci-e2e-test

LLMInferenceServiceController

stop-feature-test

Deleted

Deleted v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route
(x24)

kserve-ci-e2e-test

OpenDataHubModelController

stop-feature-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

stop-feature-test-kserve-8446b959c6-jb8f8

AddedInterface

Add eth0 [10.132.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-8446b959c6

SuccessfulCreate

Created pod: stop-feature-test-kserve-8446b959c6-jb8f8

kserve-ci-e2e-test

replicaset-controller

stop-feature-test-kserve-router-scheduler-67d85d6d77

SuccessfulCreate

Created pod: stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-router-scheduler-67d85d6d77 from 0 to 1

kserve-ci-e2e-test

deployment-controller

stop-feature-test-kserve

ScalingReplicaSet

Scaled up replica set stop-feature-test-kserve-8446b959c6 from 0 to 1

kserve-ci-e2e-test

multus

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

AddedInterface

Add eth0 [10.134.0.40/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Created

Created container: main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Started

Started container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Created

Created container: main
(x11)

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Unhealthy

Startup probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-5d74d49756-cw8kg

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-test-kserve-router-scheduler-79b8cd6b7f-l75rt

Killing

Stopping container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb
(x11)

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Unhealthy

Startup probe failed: Get "https://10.132.0.36:8000/health": dial tcp 10.132.0.36:8000: connect: connection refused
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-05aa9bba

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-74b7c95865 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

kserve-ci-e2e-test

replicaset-controller

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scheduler-74b7c95865

SuccessfulCreate

Created pod: llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-758555c5b from 0 to 1

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-rb

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-758555c5b

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

AddedInterface

Add eth0 [10.132.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

AddedInterface

Add eth0 [10.134.0.41/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Started

Started container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Started

Started container storage-initializer
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-shadow-svc

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

AddedInterface

Add eth0 [10.132.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 796ms (796ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Started

Started container tokenizer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-8446b959c6-jb8f8

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

stop-feature-test-kserve-router-scheduler-67d85d6d77-lb5bb

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.134.0.40:9003" within 1s: context deadline exceeded

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Started

Started container main
(x12)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-02ac88e7

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route" not found

kserve-ci-e2e-test

replicaset-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544

SuccessfulCreate

Created pod: llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

AddedInterface

Add eth0 [10.132.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-route

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-workload-svc

kserve-ci-e2e-test

deployment-controller

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

ScalingReplicaSet

Scaled up replica set llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Created

Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Started

Started container main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-02ac88e7

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs
(x12)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Unhealthy

Startup probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
(x12)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Unhealthy

Startup probe failed: Get "https://10.132.0.37:8001/health": dial tcp 10.132.0.37:8001: connect: connection refused

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-05aa9bba

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test4ce8108fb3d8aff064c322bee6e173f5-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-75wljrm

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schexhbwn

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7d7d5999b7f2px8

Killing

Stopping container llm-d-routing-sidecar
(x11)

kserve-ci-e2e-test

OpenDataHubModelController

custom-route-timeout-pd-test

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-79f86b944b from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-5c8d49b9d from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-prefill-787d98cd89

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-router-scheduler-79f86b944b

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

kserve-ci-e2e-test

deployment-controller

custom-route-timeout-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-787d98cd89 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

custom-route-timeout-pd-test-kserve-5c8d49b9d

SuccessfulCreate

Created pod: custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Pulled

Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Started

Started container llm-d-routing-sidecar
(x10)

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

AddedInterface

Add eth0 [10.134.0.42/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Started

Started container storage-initializer

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

AddedInterface

Add eth0 [10.132.0.40/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

AddedInterface

Add eth0 [10.132.0.41/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Created

Created container: main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs
(x9)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Unhealthy

Startup probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9d67df544-mdbgj

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Started

Started container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Created

Created container: main
(x13)

kserve-ci-e2e-test

OpenDataHubModelController

llmisvc-model-fb-opt-125m-route-f312f5ec

ReconcileError

Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-route" not found

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-scc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn successful

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-rb

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

AddedInterface

Add eth0 [10.133.0.53/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

AddedInterface

Add eth0 [10.134.0.43/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-749599699 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-scheduler-749599699

SuccessfulCreate

Created pod: llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

kserve-ci-e2e-test

statefulset-controller

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

SuccessfulCreate

create Pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0 successful

kserve-ci-e2e-test

multus

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

AddedInterface

Add eth0 [10.132.0.42/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Started

Started container storage-initializer
(x8)

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-shadow-svc

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

CreatingRevision

Creating revision with key 69454bb7bb for a newly created LeaderWorkerSet

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created leader statefulset llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn
(x2)

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Replicas are progressing, with 0 groups ready of total 1 groups

kserve-ci-e2e-test

leaderworkerset

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn

GroupsProgressing

Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-workload-svc

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Created

Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-epp-role

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Created

Created container: main

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Updated

Updated v1.Secret kserve-ci-e2e-test/llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs
(x9)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Unhealthy

Startup probe failed: Get "https://10.132.0.41:8000/health": dial tcp 10.132.0.41:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulling

Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0"

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Created

Created container: main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Started

Started container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Pulled

Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.758s (27.758s including waiting). Image size: 3531177328 bytes.

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-z7zjj

Started

Started container istio-proxy

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-prefill-787d98cd89-7lzth

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-router-scheduler-79f868l9p7

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Killing

Stopping container storage-initializer

kserve-ci-e2e-test

multus

router-gateway-2-openshift-default-6866b85949-z7zjj

AddedInterface

Add eth0 [10.133.0.54/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-z7zjj

Pulled

Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:d518f3d1539f45e1253c5c9fa22062802804601d4998cd50344e476a3cc388fe" already present on machine

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-z7zjj

Created

Created container: istio-proxy

kserve-ci-e2e-test

LLMInferenceServiceController

custom-route-timeout-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb

kserve-ci-e2e-test

gateway_labeler_controller

router-gateway-2

AddedLabel

Added label istio.io/rev=openshift-gateway to gateway router-gateway-2

kserve-ci-e2e-test

replicaset-controller

router-gateway-2-openshift-default-6866b85949

SuccessfulCreate

Created pod: router-gateway-2-openshift-default-6866b85949-z7zjj

kserve-ci-e2e-test

deployment-controller

router-gateway-2-openshift-default

ScalingReplicaSet

Scaled up replica set router-gateway-2-openshift-default-6866b85949 from 0 to 1

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuringLoadBalancer

Ensuring load balancer

kserve-ci-e2e-test

kubelet

router-gateway-2-openshift-default-6866b85949-z7zjj

Unhealthy

Startup probe failed: Get "http://10.133.0.54:15021/healthz/ready": dial tcp 10.133.0.54:15021: connect: connection refused

kserve-ci-e2e-test

service-controller

router-gateway-2-openshift-default

EnsuredLoadBalancer

Ensured load balancer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Pulling

Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1"

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

AddedInterface

Add eth0 [10.133.0.56/23] from ovn-kubernetes

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc866

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

AddedInterface

Add eth0 [10.133.0.55/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-5f6d44c486 from 0 to 1

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-prefill

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-prefill-5dfd79dc4f from 0 to 1

kserve-ci-e2e-test

deployment-controller

router-with-refs-pd-test-kserve-router-scheduler

ScalingReplicaSet

Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc866 from 0 to 1

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-5f6d44c486

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

kserve-ci-e2e-test

replicaset-controller

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f

SuccessfulCreate

Created pod: router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Started

Started container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

multus

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

AddedInterface

Add eth0 [10.134.0.44/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Started

Started container llm-d-routing-sidecar
(x9)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Created

(combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Pulled

Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 846ms (846ms including waiting). Image size: 67767940 bytes.

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Created

Created container: llm-d-routing-sidecar

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Pulled

Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0dd2798f9cdbffc0563bd148612201df0e589ff8c26a6b19d321fd120fc5c097" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Pulled

Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Pulled

Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Started

Started container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Created

Created container: tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Started

Started container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine
(x2)

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Updated

Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs
(x3)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Unhealthy

Liveness probe failed: Get "https://10.132.0.40:8000/health": dial tcp 10.132.0.40:8000: connect: connection refused
(x3)

kserve-ci-e2e-test

kubelet

custom-route-timeout-pd-test-kserve-5c8d49b9d-gvqfg

Unhealthy

Readiness probe failed: Get "https://10.132.0.40:8000/health": dial tcp 10.132.0.40:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schen8tfw

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0

Killing

Stopping container storage-initializer

kserve-ci-e2e-test

LLMInferenceServiceController

llmisvc-model-fb-opt-125m-route-f312f5ec

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9aefdf01a18c3acb55ce247710ebb71-llmisvc-mode

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

FailedToRetrieveImagePullSecret

Unable to retrieve some image pull secrets (llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-2xflz); attempting to pull the image may not succeed.
(x5)

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

FailedMount

MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs" not found

kserve-ci-e2e-test

kubelet

llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1

Killing

Stopping container main
(x9)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Unhealthy

Startup probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Pulled

Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Created

Created container: main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Started

Started container main
(x9)

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Unhealthy

Startup probe failed: Get "https://10.133.0.55:8001/health": dial tcp 10.133.0.55:8001: connect: connection refused

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-prefill-5dfd79dc4f-r9ljk

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Killing

Stopping container tokenizer

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Killing

Stopping container main

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-5f6d44c486-h2n7s

Killing

Stopping container llm-d-routing-sidecar

kserve-ci-e2e-test

LLMInferenceServiceController

router-with-refs-pd-test

Deleted

Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb

kserve-ci-e2e-test

kubelet

router-with-refs-pd-test-kserve-router-scheduler-7cdc4fc86kqzss

Unhealthy

Readiness probe failed: timeout: failed to connect service "10.134.0.44:9003" within 1s: context deadline exceeded